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PREFACE 


This  report  is  an  analysis  by  the  Human  Resources  Research  Organization  of  Instruc¬ 
tional  System  Development  (ISD)  methodology  and  of  practices  observed  in  its  application 
in  the  Army,  Navy,  Marine  Corps,  and  Air  Force.  The  research  was  conducted  under 
Contract  No.  MDA  903-77-C-0335  for  the  Office  of  the  Assistant  Secretary  of  Defense 
for  Manpower,  Reserve  Affairs  and  Logistics.  The  work  was  monitored  at  different  times 
by  Colonel  George  P.  Tilson,  Ms.  Jeannie  Fites,  and  Major  Ken  Rousseau,  all  of  OASD 
(MRA&L).  Their  assistance  is  gratefully  acknowledged. 

The  research  was  performed  by  HumRRO  Western  Division  at  Carmel,  California, 

Dr.  John  E.  Taylor,  Director.  The  Principal  Investigator  was  Dr.  Robert  Vineberg; 
Research  Associate  wate  Mr.  John  N.  Joyner,  who  contributed  to  all  phases  of  the  study 
and  writing  of  the  final  report. 

,  The  authors  wish  to  acknowledge  the  cooperation  and  assistance  of  the  many 
persons— officers,  enlisted  men,  and  civilians— who  provided  information  about  the  use 
of  ISD  in  the  military  services.  Their  number  precludes  mention  by  name.  However, 
special  mention  should  be  made  of  these  persons  who  arranged  for  the  distribution  of 
the  ISD  survey  questionnaire:  Army  -  Dr.  Joseph  Kanner,  Training  and  Doctrine  Com¬ 
mand;  Navy  -  Dr.  Worth  Scanland,  Office  of  the  Chief  of  Naval  Education  and  Training, 
Commander  Joe  Funaro,  Naval  Training  Equipment  Center,  and  Commander  Paul 
Chatelier,  Naval  Air  Systems  Command;  Marine  Corps  -  Lt.  Col.  E.A.  Grimm,  Head¬ 
quarters,  Marine  Corps;  Air  Force  -  Maj.  George  Hittle,  Air  Staff  Operations  and 
Readiness. 
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SUMMARY  AND  RECOMMENDATIONS 


PROBLEM 

The  Army,  Navy,  Marine  Corps,  and  Air  Force  all  prescribe  a  similar  sequence  of 
procedures  for  the  development  of  training.  These  procedures,  called  Instructional  Sys¬ 
tem  Development  (ISD),  are  characterized  by: 

(1)  Rigorous  derivation  of  training  requirements  from  job  requirements. 

Training  requirements  are  to  be  selected  so  as  to  Maximize  the  combined 
effectiveness  of  the  training  and  non-training  components  of  a  total 
operational  system. 

(2)  Selection  of  instructional  strategies  to  maximize  the  efficiency  of  training. 

(3)  Iterative  trial  and  revision  of  instruction  during  development  until  training 
objectives  are  met. 

ISD,  a  systems  approach  to  training  development,  has  many  potential  advantages, 
but  it  is  demanding  to  carry  out.  It  requires  sustained  commitment  to  a  repetitive 
process  of  analysis,  design,  verification,  and  revision.  Experience  in  attempts  to  insti¬ 
tutionalize  such  a  process  has  tended  to  reveal  problems. 

This  study,  performed  for  the  Office  of  the  Assistant  Secretary  of  Defense  (MRA&L), 
examined  ISD  methodologies  and  practices  in  the  four  Services  to  seek  answers  to  the 
questions: 

•  Do  current  methodologies  as  represented  in  the  major  guidance  documents 
used  in  the  Services  provide  the  means  for  attaining  the  goals  of  ISD? 

•  Do  current  applications  of  ISD  reflect  these  goals? 

•  How  can  ISD  methodologies  and  applications  be  made  mom  effective? 


APPROACH 

The  absence  of  reliable  criteria  of  system  performance  precludes  any  attempt  to 
evaluate  the  effects  of  ISD  on  performance  of  the  total  operational  military  system.  The 
study  focuses  instead  on  the  training  subsystem,  and  within  this  subsystem  it  is  restricted 
primarily  to  an  analysis  of  methodology,  process,  and  intermediate  products  as  they  bear 
on  ISD. 

The  analysis  is  based  upon  information  developed  in  three  major  activities: 

•  Analysis  of  the  primary  documents  currently  in  use  in  the  Army,  Navy, 

Marine  Corps,  and  Air  Force  for  guidance  in  conducting  ISD. 

—  Interservice  Procedures  for  Instructional  Systems  Development  (1975), 
published  by  the  Army  as  TRADOC  Pamphlet  350-30,  and  by  the  Navy 
as  NAVEDTRA  106A.  Developed  for  the  Army  and  later  approved  by 
the  Interservice  Committee  on  Instructional  Systems  Development.  Con¬ 
tains  detailed  procedures  for  conducting  ISD.  Known  as  the  ITRO  model. 

—  Marine  Corps  Order  P1510.23B  (1978).  A  greatly  reduced  version  of  the 
ITRO  model,  incorporating  documentation  requirements  specific  to  the 
Marine  Corps. 


—  Air  Force  Pamphlet  50-58,  Handbook  for  Designers  of  Instructional 
Systems  (1973-1974).  Detailed  procedures  for  implementing  the  con¬ 
cepts  of  AF  Manual  50-2,  an  overview  of  the  ISD  process. 

•  Questionnaire  survey  of  209  units,  agencies,  and  schools  where  instruction 
is  developed  in  the  four  Services. 

•  Detailed  interview  of  training  developers  at  33  of  the  organizations  surveyed 
to  determine  how  57  specimen  courses  were  designed. 

A  generalized  ISD  model  composed  of  19  training  development  steps  was  used  to 
provide  a  structure  for  analyzing  and  organizing  the  information  obtained,  and  summarizing 
findings  about  ISD  practices.  The  steps  and  requirements  of  this  generalized  ISD  model 
were  derived  in  part  from  synthesizing  the  Service  models,  and  in  part  through  rational 
analysis  of  the  intended  functions  of  those  models. 


GENERAL  FINDINGS  AND  RECOMMENDATIONS 

Judgments  about  the  adequacy  of  Instructional  System  Development,  its  representa¬ 
tion  ir.  the  Service  models,  and  the  way  it  is  currently  being  applied  depend  on  one’s 
conception  of  what  ISD  is  and  what  it  is  expected  to  accomplish. 

The  most  general  way  to  define  ISD  is  as  a  means  of  designing  training  to  optimize 
total  system  effectiveness.  Criteria  do  not  exist  for  measuring  total  system  effectiveness; 
ISD  is  not  being  used  to  achieve,  nor  can  it  be  expected  to  result  in,  optimizing  the 
effectiveness  of  the  total  system. 

A  more  circumscribed  view  of  ISD  is  as  a  methodology  for  maximizing  training 
efficiency  within  the  training  subsystem.  However,  information  about  the  effectiveness 
and  costs  of  different  training  strategies  is  far  from  complete,  and  a  trial -and -error 
approach  to  maximizing  efficiency  is  not  practical,  given  the  number  of  possible  combi¬ 
nations  of  methods.  While  ISD  does  provide  a  framework  for  comparing  alternative 
training  strategies,  it  is  not  currently  being  used,  nor  can  it  be  regarded,  as  a  methodology 
for  maximizing  training  efficiency. 

A  more  highly  focused  view  is  that  ISD  is  a  methodology  for  insuring  that  training 
is  relevant  to  the  job.  Its  iterative  and  derivative  character  virtually  assures  that  training 
will  be  relevant  if  available  procedures  are  faithfully  carried  out.  In  practice,  however, 
many  of  its  components  are  omitted,  and  the  close  connection  between  components  that 
is  essential  to  make  the  process  truly  derivative  is  not  maintained.  Most  important,  the 
testing  and  revision  necessary  to  insure  job  relevance  generally  do  not  occur.  The  potential 
of  ISD  to  insure  that  training  meets  job  requirements  is  not  being  realized. 

A  final  conception  of  ISD  is  that  it  is  synonymous  with  the  use  of  modern  training 
technology.  Any  of  the  steps  in  the  training  development  process  are  a  part  of  modem 
training  technology,  and  so  are  any  of  those  particular  training  and  evaluation  methods 
currently  being  emphasized  (e.g.,  self-paced  instruction,  computer -managed  instruction, 
criterion-referenced  testing).  This  definition  of  ISD  is  clearly  the  least  demanding,  since 
in  essence  it  holds  that  undertaking  any  training  development  step  or  using  any  such 
training  or  evaluation  method  constitutes  ISD.  It  is  the  definition  that  is  most  adequately 
represented  in  current  applications  of  ISD. 
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In  summary,  then,  two  effects  of  ISD  are  currently  possible:  insuring  that  training 
meets  job  requirements  and  promoting  the  use  of  modem  training  technology.  The  former, 
which  is  clearly  the  more  desirable,  is  not  being  achieved.  The  ISD  modes  does  provide 
the  methodology  for  making  training  relevant,  but  the  mere  existence  of  the  model  does 
not  compel  trainers  to  follow  it.  Trainers  are  relatively  free,  within  fairly  broad  limits, 
to  determine  the  extent  to  which  they  will  conform  to  the  ISD  process  and  actually  use 
its  products  in  designing  training.  For  example,  front-end  analysis  may  be  undertaken, 
but  its  results  can  be  and  frequently  are  ignored.  Training  objectives  often  are  devel¬ 
oped  with  indifference  to,  or  in  ignorance  of,  actual  task  requirements.  Many  tests  and 
instructional  materials  are  developed  without  regard  to  training  objectives.  Information 
about  the  performance  of  trainees,  once  collected,  is  often  not  used  to  revise  training, 
and  feedback  about  graduates,  if  initiated,  is  often  not  acted  upon. 

Because  ISD  is  a  process,  it  is  difficult  to  observe  directly.  The  occurrence  of  a 
process  is  generally  inferred  from  the  presence  or  absence  of  its  products.  In  the  case 
of  ISD,  however,  the  mere  existence  of  its  products— job  task  lists,  training  objectives, 
achievement  tests,  and  the  like— does  not  by  itself  indicate  that  they  have  been  used  in 
training  development.  Considerable  evidence  that  many  ISD  products  remain  unused 
leads  to  the  conclusion  that,  while  the  generation  of  these  products  can  be  mandated, 
the  ISD  process  and  the  appropriate  use  of  the  products  during  training  development 
cannot. 

Similarly,  a  routine  allocation  of  responsibilities  in  ISD  does  not  necessarily  guar¬ 
antee  that  these  responsibilities  will  be  recognized,  accepted,  and  carried  out.  For  example, 
ISD  methodology  requires  developers  to  specify  the  way  in  which  proficiency  will  be 
developed  or  mediated  for  all  job  tasks— immediately,  through  entry  training;  later, 
through  advanced  training  or  job  experience;  or  through  direct  support  of  job  performance 
by  procedural  aids  or  other  means.  Yet  this  requirement  is  frequently  ignored.  ISD  is 
generally  conducted  to  develop  training  for  only  one  particular  setting,  and  the  manner  in 
which  skill  will  be  acquired  for  tasks  that  are  not  selected  for  training  in  that  setting  is 
usually  not  specified.  Even  where  skills  were  explicitly  identified  for  later  acquisition, 
the  present  study  found  little  evidence  either  that  means  were  developed  for  subsequent 
training  or  that  operational  units  were  informed  of  their  responsibilities  for  insuring  that 
these  skills  be  acquired.  (This  observation  is  based  on  training  development  within  the 
training  subsystem.  The  present  study  did  not  examine  training  developed  or  conducted 
in  operational  units.  Possibly  such  an  examination  would  indicate  that  training  for  job 
skills  in  these  units  is  being  conducted  in  a  more  comprehensive  manner  than  was  sug¬ 
gested  by  the  evidence  in  the  present  study.) 

The  current  failure  of  ISD  applications  to  insure  that  training  meets  job  requirements, 
then,  is  largely  due  not  to  inadequacies  in  the  methodology,  but  to  omissions  and  to 
failure  to  use  its  products  in  a  way  that  makes  the  process  truly  derivative.  An  implica¬ 
tion  of  these  findings  is  that  future  efforts  to  implement  ISD  should  concentrate  on 
finding  ways  to  maintain  the  integrity  of  the  model. 

The  findings  of  the  study  do  not  of  themselves  indicate  how  to  assure  rigorous 
adherence  to  what  is  clearly  a  very  demanding  model,  but  they  do  suggest  that  intro¬ 
ducing  changes  solely  within  the  training  subsystem  is  not  likely  to  have  any  great  effect. 
The  data  strongly  suggest  the  need  for  checks  and  balances  to  guard  against  omissions  in 
the  ISD  process  and  failures  to  use  the  ISD  products.  It  would  appear  logical,  therefore, 
to  provide  for  an  expanded  role  by  operational  commands— the  party  directly  affected 
by  shortcomings  in  training,  and  best  able  to  assess  the  effects  of  training. 
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Such  an  expanded  role  for  operational  commands  is,  in  fact,  implied  by  the  model 
itself.  The  derivative  and  iterative  aspects  of  ISD  depend  on  feedback  and  exchange  of 
information  between  trainers  and  users.  It  is  difficult  to  see  how  ISD  could  be  more 
rigorously  applied  unless  such  an  exchange  takes  place.  In  principle,  the  training  sub¬ 
system  seeks  information  about  field  requirements  and  performance  as  the  foundation  on 
which  training  is  constructed.  In  practice,  however,  the  study  found  that  the  training 
subsystem  does  not  have  this  basic  orientation,  often  giving  insufficient  attention  to  the 
effective  use  of  this  information.  It  is  reasonable  to  suggest  that  a  balanced  relationship- 
one  that  fosters  active  participation  by,  and  communication  between,  operational  and 
training  commands— is  essential  to  the  ISD  process. 

The  following  recommendations  define  the  means  by  which  operational  commands 
can  assume  a  greater  role.  Under  these  recommendations,  operational  commands  would 
participate  to  the  greatest  extent  in  those  parts  of  ISD  where  job  performance  is  repre¬ 
sented  and  where  job  performance  requirements  are  translated  into  training  requirements. 
Those  parts  of  ISD  concerned  with  the  design  of  instructional  strategies  to  meet  train¬ 
ing  and  job  requirements  would  remain  the  province  of  the  training  community. 

It  is  recommended: 

(1)  That  job  requirements  (skills  and  knowledge  required  for  successful  job 
performance)  be  jointly  defined,  and  agreed  to,  by  training  and  opera¬ 
tional  commands.  This  recommendation  is  a  prerequisite  to  Recommenda¬ 
tion  2,  and  to  all  subsequent  training  development  activities.  If  successful 
dialogue,  negotiation,  and  agreement  are  to  follow,  training  requirements 
must  be  based  on  a  mutually  agreed-upon  definition  of  job  requirements. 

(2)  That  training  requirements  (skills  and  knowledge  to  be  available  at  the 
conclusion  of  training)  be  jointly  defined,  and  agreed  to,  by  training 

and  operational  commands.  Such  an  agreement  should  identify  the  specific 
tasks  and  standards  to  which  proficiency  will  be  developed,  and  should 
delineate  the  respective  responsibilities  of  the  two  parties.  This  includes 
providing  a  means  for  bringing  job  incumbents  to  the  desired  level  of 
proficiency  whenever  agreed-upon  training  requirements  do  not  match  job 
requirements. 

(3)  That  operational  commands  be  required  to  evaluate  the  performance  of 
training  graduates,  and  report  their  findings  to  the  training  commands. 
Unless  operational  commands  evaluate  performance,  feedback  from  users 
to  trainers  will  not  have  a  sound  basis.  Without  reliable  information  about 
the  effects  of  training,  specification  of  training  requirements  will  not 
serve  its  purpose. 

(4)  To  implement  Recommendation  3,  that  task-specific  criteria  for  evaluating 
the  performance  of  graduates,  including  methods  and  standards  to  be 
employed,  be  jointly  defined,  and  agreed  to,  by  training  and  operational 
commands.  Evaluation  criteria  should  be  at  the  task  level  of  specificity 
to  permit  clear  and  useful  diagnosis  of  training.  More  general  evaluations 
are  of  little  use  in  isolating  the  causes  of  inadequate  performance.  Eval¬ 
uation  criteria  must  be  jointly  agreed  to  if  the  results  of  evaluations  are 
to  be  accepted  as  valid,  and  acted  upon. 


Giving  l he  operational  commands  a  greate*  role  in  both  establishing  training  require¬ 
ments  and  determining  whether  requirements  Lave  been  met  will  not  of  itself  guarantee 
that  ISD  procedures  will  be  rigorously  applied  to  the  development  of  training.  It  will, 
however,  increase  the  involvement  of  those  who  have  the  most  fundamental  interest  in 
seeing  that  training  has  been  adequately  designed  and  conducted. 

So  long  as  training  development  and  evaluation  are  regarded  as  a  separate  activity 
of  the  training  commands,  there  is  little  reason  to  expect  that  ISD  will  be  applied  any 
more  effectively  than  under  the  present  conditions.  If  training  and  operational  commands 
share  these  activities— each  making  its  specialized  contribution  to  complement  the  work 
of  the  other— the  potential  of  the  ISD  process  for  improving  training  will  be  enhanced. 


FINDINGS  AND  RECOMMENDATIONS  FOR 

SPECIFIC  ISD  STEPS 

Need/Discrepancy  Analysis 

Findings 

1.  Methodology  -  Present  ISD  models  do  not  include  procedures  for  identifying  problems 
in  existing  training  and  for  identifying  a  need  to  undertake  ISD.  Guidance  is  needed 
on  how  to  identify  discrepancies  between  existing  training  and  the  field  requirements 
for  a  job,  and  how  to  revise  training  short  of  undertaking  the  entire  ISD  process. 

2.  Application  -  ISD  is  not  generally  initiated  in  response  to  specific  discrepancies 
between  training  and  field  requirements. 

3.  Application  -  ISD  is  usually  initiated  in  response  to  a  directed  change  (e.g., 
provide  individualized  and  self-paced  instruction)  or  to  a  requirement  to  revise 
existing  training  in  accordance  with  ISD  methods. 

Recommendations 

1.  Where  existing  training  is  being  examined,  ISD  methodologies  should  emphasize 
evaluating  and  improving  the  training,  rather  than  simply  assuming  that  development 
of  a  new  course  is  appropriate.  At  present  new  courses  are  rarely  evaluated  any 
more  rigorously  than  the  ones  they  replace. 

2.  To  increase  the  emphasis  on  evaluation  and  improvement,  specific  procedures  should 
be  developed  both  to  identify  faults  in  existing  training  and  to  determine  efficient 
boundaries  for  the  ISD  process. 


IDENTIFICATION  OF  JOB  REQUIREMENTS 

Findings 

1.  Methodology  -  Current  guidance  for  identifying  job  requirements  permits  con¬ 
siderable  latitude  in  the  approach  taken  and  the  level  of  description  used,  with 
resulting  variation  in  the  reliability  and  utility  of  the  information  developed. 


2.  Application  -  Job  analysis  is  usually  confounded  with  the  selection  of  tasks  for 
training.  Emphasis  is  not  given  to  independently  specifying  requirements  as  they 
exist  in  the  job. 

3.  Application  -  Job  task  lists  from  occupational  surveys  (e.g.,  CODAP,  NOTAP)  are 
sometimes  available.  The  information  provided  by  such  lists,  however,  is  often  in 
part  about  classes  of  activities  rather  than  about  tasks,  and  to  that  extent  may  not 
serve  as  an  adequate  base  for  deriving  training.  Often,  complete  task  lists  are  not 
developed. 

Recommendation 

1.  Training  developers  should  be  required  to  provide  and  maintain  a  description  of  job 
task  requirements  distinct  from  a  listing  of  tasks  selected  for  training.  This  would 
make  explicit  the  extent  to  which  training  requirements  differ  from  job  :  ■quirements. 


Selection  of  Tasks  for  Training 

Findings 

1.  Methodology  -  There  are  no  measures  of  system  effectiveness  that  can  be  used  to 
validate  criteria  for  selecting  tasks  for  training,  and  rules  for  applying  such  criteria. 

As  a  result,  the  choice  of  criteria  to  be  used  in  selecting  tasks  to  be  trained  is  left 
to  personal  judgment. 

2.  Application  -  Selecting  tasks  for  training  is  generally  not  preceded  by  a  separate 
and  distinct  delineation  of  the  tasks  required  by  the  job. 

3.  Application  -  Task  selection  for  training  is  usually  not  done  systematically.  Its 
rationale  is  rarely  explicit. 

4.  Application  -  Task  priorities  (that  would  provide  the  basis  for  getting  the  maximum 
training  benefits  from  the  available  funding)  are  not  specified. 

Recommendations 

1.  Training  developers  should  be  required  to  make  explicit  the  basis  on  which  they 
select  tasks  for  training,  and  to  specify  priorities  among  both  the  tasks  selected  and 
the  tasks  rejected  for  training  for  a  particular  job. 

2.  In  the  absence  of  information  about  the  effects  of  task  selection  criteria  on  system 
performance,  guidance  should  be  developed  on  the  types  of  task  priority  information 
likely  to  be  relevant  for  different  classes  of  jobs. 


Analysis  of  Tasks 

Findings 

1.  Methodology  -  Procedures  for  analyzing  tasks  are  adequate. 

2.  Application  -  Job  task  conditions  and  standards,  as  distinct  from  training  conditions 
and  standards,  are  seldom  identified.  Course  control  documents  do  not  require  that 
job  characteristics  be  specified. 


Recommendation 

1.  Benefits  of  attempting  to  modify  current  practices  do  not  appear  great  relative  to 
costs.  No  change  in  present  practice  is  recommended. 

Construction  of  Job  Performance  Measures 

Findings 

1.  Methodology  •  Procedures  for  developing  JPMs  are  adequate  in  the  1TRO  model, 
unclear  in  the  Marine  Corps  model,  and  not  included  in  the  AFP  50-58  model. 

2.  Application  -  JPMs  have  been  developed  in  jobs  where  the  consequences  of  inade¬ 
quate  performance  are  especially  serious,  such  as  in  flight  training  or  use  of  special 
weapons. 

3.  Application  -  In  only  one  instance  were  JPMs  developed  as  part  of  ISD  to  validate 
within-course  tests. 

Recommendation 

1.  While  theoretically  worthwhile,  JPMs  are  costly  to  develop.  It  is  unrealistic  to 
recommend  their  development  except  in  special  instances.  No  change  in  present 
practice  is  recommended. 

Selection  of  Setting 

Findings 

1.  Methodology  -  A  systematic  procedure  for  determining  the  optimal  setting  for 
training  does  not  exist.  The  development  of  such  a  procedure  waits  upon  a  means 
for  measuring  system  performance,  which  in  turn  will  permit  the  validation  of  site 
selection  criteria  and  decision  rules. 

2.  Application  -  In  general,  training  developers  do  not  have  the  authority  to  designate 
and  develop  training  in  different  settings. 

Recommendation 

1.  The  choice  of  training  setting  has  effects  on  the  operational  system  beyond  the 
training  subsystem.  It  is  appropriate  that  the  setting  selection  be  made  at  a  higher 
level  than  training  developer.  In  the  absence  of  a  means  for  assessing  total  system 
effectiveness,  no  change  in  present  practice  is  recommended. 

Development  of  Training 
Objectives  and  Objectives  Hierarchies 

Findings 

1.  Methodology  -  The  ITRO  model  provides  the  most  comprehensive,  explicit,  and 
straightforward  procedures  for  translating  job  requirements  into  training  objectives. 


Deficiencies  in  the  Marine  Corps  model  could  encourage  writing  objectives  to  match 
what  is  being  taught,  rather  than  to  meet  job  requirements. 

2.  Application  -  The  specification  of  training  objectives  is  virtually  universal,  but  the 
procedures  used  to  identify  objectives  are  highly  variable  and  frequently  unclear. 
There  is  evidence  that  objectives  are  often  prepared  after  the  fact  and  are  derived 
from  training  content  rather  than  used  to  generate  it. 

3.  Application  -  Terminal  objectives  are  seldom  subjected  to  an  explicit  hierarchical 
analysis  to  derive  intermediate  objectives.  A  determination  of  the  skills  and  knowl¬ 
edge  that  would  enable  the  trainee  to  meet  the  terminal  objectives  would  emerge 
from  such  an  analysis;  these  often  are  not  identified. 

4.  Application  -  Even  where  records  are  maintained,  formats  for  displaying  the  relation 
between  tasks  and  training  objectives  make  it  hard  to  determine  what  objectives 
have  been  derived  from  a  given  task.  That  is,  tasks  that  represent  objectives  are 
displayed,  rather  than  objectives  that  have  been  derived  from  each  task.  Thus,  the 
justification  for  training  for  specific  objectives  is  often  not  clear. 

Recommendation 

1.  The  derivation  of  training  objectives  from  job  tasks  should  be  made  explicit  in  a 
format  that  cross-references  objectives  by  task. 


Development  of  Achievement  Tests 


Findings 

1.  Methodology  -  All  of  the  models  require  that  achievement  tests  be  developed  from 
training  objectives,  rather  than  from  the  content  of  lessons,  and  all  provide  some 
information  about  test  construction.  All  models  lack  procedures  for  maintaining 
congruence  between  the  behaviors  implied  in  an  objective  and  the  actual  require¬ 
ments  imposed  by  test  items  (e.g.,  use  performance  tests  to  measure  skilled  behavior, 
require  that  concepts  be  applied  when  an  objective  implies  their  use  rather  than 
their  recall  or  recognition). 

2.  Application  -  Many  achievement  tests  are  derived,  not  directly  from  training  objec¬ 
tives,  but  from  training  content.  Knowledge  tests  are  particularly  likely  to  be 
derived  from  content.  In  these  cases,  no  independent  criterion  exists  to  deter¬ 
mine  whether  training  objectives  have  been  met. 

3.  Application  -  In  general,  little  or  no  consideration  is  given  to  matching  the  type  and 
level  of  test  items  to  the  behavioral  requirements  of  objectives  (see  Finding  1). 

Recommendations 

1.  ISD  models  should  explain  and  emphasize  the  purpose  and  need  for  deriving 
achievement  tests  from  training  objectives  rather  than  from  training  content. 

2.  ISD  models  should  be  expanded  to  provide  procedures  for  identifying  and  main¬ 
taining  congruence  between  the  behavioral  requirements  of  objectives  and  test  items. 
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Identification  of  Entry  Behavior 


Findings 

1.  Methodology  -  The  major  1SD  models  provide  procedures  for  adjusting  training 
objectives  to  match  trainee  capabilities. 

2.  Application  -  Estimates  of  capabilities  of  trainee  populations  are  not  verified  by 
testing  before  the  training  is  implemented. 

Recommendation 

1.  The  advantages  of  correcting  inaccurate  estimates  of  trainee  capabilities  do  not 
appear  to  justify  the  costs  of  the  measurement  that  would  be  required.  Over¬ 
estimates  are  likely  to  be  identified  during  the  validation  of  instruction,  and  under¬ 
estimates  usually  become  evident  during  the  conduct  of  instruction.  No  change  in 
present  practice  is  recommended  unless  large  investments  in  instructional  materials 
are  involved  (e.g.,  Training  Extension  Course  (TEC)  development). 


Classification  of  Objectives  and 
Selection  of  instructional  Activities 


Findings 

1.  Methodology  -  Procedures  for  classifying  training  objectives,  and  for  selecting  instruc¬ 
tional  activities  accordingly,  are  not  highly  developed.  Different  models  use  different 
taxonomies  for  classifying  objectives,  and  guidance  for  both  classifying  objectives 
and  selecting  instructional  activities  is  provided  largely  by  example  rather  than  by 
means  of  explicit  decision  rules. 

2.  Application  -  Training  objectives  generally  are  not  classified,  and  instructional  activ- 
ities  generally  are  not  specified. 

Recommendation 

1.  Explicit  decision  rules  for  classifying  objectives  and  selecting  instructional  activities 
should  be  developed.  In  the  absence  of  such  rules,  no  change  in  present  practice 
is  recommended. 


Selection  of  Instructional  Methods 


Findings 

1.  Methodology  -  All  models  describe  and  discuss  alternative  instructional  methods. 

They  specify  prerequisite  conditions  (e.g.,  setting,  group  size)  for  the  use  of  particu¬ 
lar  methods  but  provide  little  information  about  the  relative  effectiveness  of  different 
methods,  either  for  particular  types  of  content  or  for  trainee  populations.  Though 
perhaps  sufficient  to  allow  the  training  developer  to  reject  inappropriate  methods, 
the  information  base  and  the  models  themselves  are  not  sufficient  to  provide  for 
selecting  optimal  methods. 
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2.  Application  -  Training  methods  are  not  systematically  selected  either  on  the  basis 
of  instructional  activities  (which  are  also  not  specified)  or  on  the  basis  of  trainee 
characteristics. 

3.  Application  -  Changes  in  training  methods  are  almost  always  in  response  to  com¬ 
mand  policy. 

Recommendation 

1.  Present  training  technology  is  not  advanced  enough  to  support  proceduralized 
derivation  of  training  methods  from  previously  specified  instructional  activities 
and  trainee  characteristics.  Information  about  optimal  training  methods  for  dif¬ 
ferent  training  situations,  and  procedures  to  enable  developers  to  identify  the  most 
promising  methods,  should  be  developed.  In  the  absence  of  such  procedures,  no 
change  in  present  practice  is  recommended. 


Selection  of  Media 


Findings 

1.  Methodology  -  Procedures  for  selecting  training  media  appear  adequate  to  match 
the  presentation  (stimulus  and  response)  requirements  of  instructional  activities  to 
appropriate  media,  if  instructional  activities  have  been  specified  in  sufficient  detail. 

2.  Application  -  Training  media  are  not  systematically  selected  on  the  basis  of  require¬ 
ments  of  instructional  activities  (which  are  also  not  specified). 

3.  Application  -  Developers  generally  do  not  have  freedom  to  select  among  alternative 
media.  Choices  and  changes  in  media  are  usually  directed  by  command  policy. 

Recommendation 

1.  Selection  of  appropriate  media  is  contingent  on  how  well  instructional  activities 
have  been  specified.  No  change  in  the  current  models  for  matching  media  to 
activities  is  appropriate  until  activities  are  more  widely  specified,  and  these  models 
can  be  tested. 


Grouping  and  Sequencing  of  Instruction 


Findings 

1.  Methodology  -  Universal  principles  for  grouping  and  sequencing  instructional  objec¬ 
tives  do  not  exist,  other  than  that  dependent  objectives  be  placed  later  than  those 
on  which  they  depend.  Alternate  theories  and  conflicting  strategies  abound,  with 
no  knowledge  base  for  resolving  them.  Because  systematically  related  principles 
for  promoting  learning  are  lacking,  sequencing  instruction  must  be  left  to  the 
individual  judgment  of  the  training  developers. 

2.  Application  -  Practices  in  grouping  and  sequencing  instruction  generally  give  no  con¬ 
sideration  to  learning  requirements  other  than  the  order  imposed  by  obvious  depend¬ 
encies.  Attention  is  given  to  constraints  of  non-learning  factors  such  as  equipment 
availability  and  scheduling. 
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Recommendation 


1.  Organized  and  systematically  related  principles  of  learning  on  which  to  base  grouping 
and  sequencing  decisions  are  not  available.  The  development  of  such  information 
falls  within  the  psychology  of  learning  and  is  beyond  the  scope  of  recommendations 
in  this  report.  In  the  absence  of  such  information,  no  change  in  present  practices 
is  recommended. 


Development  of  Plan  for 
Authoring  and  Managing  Instruction 


Findings 

1.  Methodology  -  All  models  require  the  preparation  of  a  plan  for  authoring  and 
managing  instruction. 

2.  Application  -  Plans  that  record  course  content  are  often  prepared. 

3.  Application  -  Plans  typically  do  not  specify  instructional  events  and  are  rarely  used 
to  develop  instructional  materials. 

Recommendation 

1.  Explicit  decision  rules  for  selecting  instructional  activities  do  not  presently  exist 
(see  Classification  of  Objectives  and  Selection  of  Instructional  Activities  above). 
Until  such  rules  are  available  and  generally  acknowledged  as  valid,  a  requirement  to 
specify  instructional  activities  in  an  authoring  and  managing  plan  is  likely  to  be 
viewed  as  a  pointless  exercise.  No  change  in  present  practice  is  recommended. 


Review  and  Selection  of  Existing  Materials 


Findings 

Methodology  -  The  ITRO  and  Marine  Corps  models  state  that  decisions  to  use 
existing  materials  (rather  than  author  new  instruction)  are  to  be  based  on  the 
appropriateness  of  these  materials  to  the  previously  specified  characteristics  of 
objectives,  methods,  and  media.  The  AFP  50-58  model  does  not  provide  guidance 
for  reviewing  and  selecting  existing  material. 

2.  Application  -  Characteristics  to  be  identified  in  judging  the  appropriateness  of 
existing  training  materials  are  not  specified. 

3.  Application  -  Review  and  use  of  existing  training  materials  is  minimal,  except  for 
those  in  a  course  that  is  being  revised. 

Recommendation 

1.  The  specification  of  necessary  properties  of  materials  for  particular  training  situa¬ 
tions,  and  the  description  and  cataloging  of  existing  materials  to  permit  the  inter¬ 
change  of  matching  components  across  courses,  represent  a  degree  of  perfection 
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that  is  not  presently  attainable.  An  attempt  to  reduce  the  review  and  selection  of 
existing  materials  to  a  systematic  procedure  is  to  act  as  if  the  methods  of  a  well- 
developed  technology  were  available  in  an  area  in  which  judgment  is  in  fact  the 
dominant  factor.  No  change  in  present  practice  is  recommended. 

Authoring  of  Instruction 

Findings 

1.  Methodology  -  All  models  specify  that  instructional  materials  undergo  tryout  during 
the  authoring  process.  The  ITRO  and  Marine  Corps  models  emphasize  that  instruc¬ 
tion  should  be  lean  to  insure  the  economies  of  minimal  instruction. 

2.  Application  -  There  is  little  awareness  of  the  concept  of  lean  instruction,  and 
few  attempts  to  develop  it. 

3.  Application  -  Instruction  is  rarely  given  tryout  and  revision  during  authoring. 

Recommendation 

1.  Training  managers  should  receive  guidance  on  the  purpose  and  importance  of 
developing  lean  instruction.  Guidance  should  indicate  the  role  of  tryout  and 
revision  of  instruction  as  a  necessary  element  of  this  strategy. 

Validation  of  Instruction 

Findings 

1.  Methodology  -  All  models  specify  satisfactory  procedures  for  validating  instruction. 
The  adequacy  of  training  materials  for  attaining  objectives  is  verified  through  the 
administration  of  achievement  tests. 

2.  Application  -  Validation  criteria— that  is,  evidence  that  instruction  is  satisfactory- 
are  rarely  specified. 

3.  Application  -  Instruction  is  rarely  validated  before  it  is  implemented.  When  vali¬ 
dation  does  occur,  it  is  training  materials  (e.g.,  textbooks,  tape/slide  programs) 
that  are  evaluated;  instructor  lessons  and  lesson  plans  are  almost  never  evaluated. 

Recommendation 

1.  Of  the  three  major  types  of  ISD  evaluation  (validation,  internal  evaluation,  external 
evaluation),  validation  has  the  greatest  potential  for  effecting  improvements  in 
instruction.  Once  instructional  materials  have  been  produced  and  instruction  has 
been  implemented,  changes  are  less  likely  to  be  introduced,  and  new  materials  are 
more  difficult  to  generate.  Validation  trials  to  meet  specified  criteria  should  be 
required  before  new  instruction  is  approved. 


Internal  Evaluation 


Findings 

1.  Methodology  -  All  models  specify  adequate  procedures  for  the  internal  evaluation 
of  training.  Quality  control  of  the  training  product  is  to  be  maintained  through 
the  administration  of  objective-referenced  achievement  tests. 

2.  Application  -  Evaluation  and  revision  of  instruction  based  on  needs  revealed  in 
student  performance  (product  evaluation)  are  generally  not  done. 

3.  Application  -  Training  design  decisions  are  rarely  documented  (process  evaluation), 
to  facilitate  redesign  when  instruction  is  found  to  be  inadequate. 

Recommendation 

1.  Trainers  should  be  required  to  determine  and  record  trainee  performance  for  each 
objective.  Although  absolute  standards  to  identify  when  training  revision  is  needed 
are  difficult  to  establish,  the  recording  of  specific  trainee  performance  would  pro¬ 
vide  a  desirable  prerequisite  to  any  revision.  Moreover,  it  would  suggest  relative 
standards  for  the  need  to  revise  training. 


External  Evaluation 


Findings 

Methodology  -  the  1TRO  model  prescribes  the  most  reliable  and  most  costly  way 
to  measure  the  adequacy  of  the  instructional  design  process:  administering  Job 
Performance  Measures  to  graduates  in  the  field.  It  also  provides  the  most  guidance 
for  isolating  causes  of  performance  discrepancies  after  the  external  evaluation  of 
training.  If  summary  evaluations  are  to  be  used,  the  ITRO  and  AFP  50-58  models 
specify  that  information  be  obtained  at  the  task  level  of  specificity,  while  the 
Marine  Corps  model  does  not. 

2.  Methodology  -  None  of  the  models  specify  criteria  that  should  be  used  to  determine 
whether  training  is  to  be  revised  as  a  result  of  external  evaluation,  nor  how  to 
arrive  at  such  criteria.  None  tell  how  good  job  performance  must  be  to  indicate 
that  training  is  acceptable. 

3.  Application  -  The  effectiveness  of  training  is  virtually  never  evaluated  by  ti  e  admin¬ 
istration  of  Job  Performance  Measures  to  job  incumbents. 

4.  Application  -  Supervisor  summary  evaluations  of  job  incumbent  performance  are 
occasionally  obtained  but  usually  are  not  provided  at  a  task  or  training  objective 
level  of  specificity.  Even  when  performance  and  job  requirements  information  is 
obtained,  it  is  rarely  used  to  redesign  training. 


xvii 


Recommendations 


1.  The  high  cost  of  administering  performance  tests  to  job  incumbents  and  the  diffi¬ 
culty  of  maintaining  the  necessary  degree  of  objectivity  and  standardization  pre¬ 
clude  their  widespread  use  to  evaluate  training.  No  change  in  present  practice  is 
recommended. 

2.  Training  can  be  evaluated  by  gathering  information  (rather  than  direct  measurement) 
about  job  performance  and  job  requirements,  at  a  task  and  training  objective  level 
of  specificity,  far  more  thoroughly  than  is  presently  done.  The  failure  to  obtain 
and  use  such  information  is  a  major  shortcoming  in  current  applications  of  ISD. 
While  decision  rules  for  using  such  information  cannot  at  present  be  based  on 
measures  of  total  system  effectiveness,  other  means  for  a.Tiving  at  such  criteria 

are  available.  It  is  recommended  that  operational  commands  define  both  the 
specificity  of  the  task  description  and  the  level  of  performance  they  would  be 
willing  to  use  to  evaluate  the  acceptability  of  job  incumbents.  These  are  the 
criteria  that  supervisors  should  use  to  judge  (rate)  field  performance  and  establish 
the  need  for  training  revision. 
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Chapter  1 

INTRODUCTION 


SYSTEMS  ANALYSIS  AND  TRAINING  DEVELOPMENT 

Beginning  in  the  late  1950s,  systematic  procedures  began  to  be  applied  to  the  design 
of  training  programs  in  the  military  services.  Early  methodologies,  influenced  in  general 
by  operations  analysis  concepts  of  World  War  II  and  specifically  by  the  recognition  of  a 
need  for  requirements  analysis,  emphasized  empirical  determination  of  the  requirements 
of  jobs  and  clear  specification  of  the  objectives  of  training. 

During  the  1960s  and  early  1970s,  instructional  design  procedures  became  more  codi¬ 
fied.  Models  prescribing  specific  sequences  of  training  development  emerged.  To  the 
“front-end”  analysis  of  earlier  procedures,  these  later  models  added  steps  for  the  design 
and  development  of  instructional  content,  as  well  as  its  implementation  and  control.1 

Today  the  Army,  Navy,  Marine  Corps,  and  Air  Force  all  prescribe  a  similar  sequence 
of  procedures  for  the  development  of  training.  For  example,  all  of  the  Services  require 
that  the  design  of  training  begin  with  an  analysis  of  job  requirements.  All  prescribe  that 
these  requirements  be  stated  in  terms  of  capabilities  to  perform  specific  tasks,  that  training 
objectives  be  derived  from  these  tasks,  that  instructional  content  be  selected  solely  to 
meet  the  objectives,  and  that  instructional  events  be  determined  by  characteristics  of  the 
content,  capabilities  of  trainees,  and  level  of  mastery  to  be  attained.  None  of  the  meth¬ 
odologies  permit  a  rearrangement  of  the  process  so  as  to,  for  example,  allow  objectives 
to  be  determined  by  already  existing  instructional  materials. 

These  procedures,  though  differing  somewhat  in  organization  and  detail  both  across 
and  within  Services,  are  all  models  of  essentially  the  same  process  and  are  currently 
referred  to  as  Instructional  System  Development  (ISD).2  The  process  is  characterized  by 
a  number  of  features  common  to  systems  analysis  in  general. 

•  A  system  is  comprised  of  subsystems  and  their  interfaces.  In  ISD,  training 
is  recognized  as  a  subsystem  of  a  larger  operational  system  and  interfaces 
with  other  subsystems  (e.g.,  weapon  system  and  job  requirements,  per¬ 
sonnel  classification  and  assignment  policies).  Training  design  decisions  during 
the  ISD  process  are  made  with  a  view  toward  optimizing  the  operation  of 
both  the  total  operational  system  and  the  training  subsystem. 


1  Earlier  versions  of  these  procedures  included:  Design  of  Instructional  Systems  (DIS),  Systems 
Approach  to  Training  (SAT),  Systems  Engineering  of  Training  (SET),  and  Training  Situation  Analysis 
(TSA).  For  a  historical  review  of  these  methodologies  and  the  development  of  the  systems  approach 
to  training,  see  Montemerlo,  Melvin  D.  and  Tennyson,  Michael  E.,  Instructional  Systems  Development: 
Conceptual  Analysis  &  Comprehensive  Bibliography,  Technical  Report  NAVTRAEQUIPCEN  1H-257, 
Human  Factors  Laboratory,  Naval  Training  Equipment  Center,  February  1976. 

2 The  ISD  models  in  use  in  the  Armed  Services  are  described  in  a  variety  of  documents  and 
manuals.  Primary  documents  currently  include  Air  Force  Manual  50-2,  Instructional  System  Development 
(1970);  Air  Force  Pamphlet  50-58,  Handbook  for  Designers  of  Instructional  Systems  (1973,  1974); 

Marine  Corps  Order  P1510.23B  (1978);  Interservice  Procedures  for  Instructional  Systems  Development 
(1975)  (published  by  the  Army  as  TRADOC  Pamphlet  350-30  and  by  the  Navy  as  NAVEDTRA  106A). 
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•  Subsystem  objectives  and  configurations  are  based  on  an  analysis  of  system 

requirements  and  subsystem  relationships.  The  ISD  process  begins  with  a 
detailed  analysis  of  job  requirements  and  a  determination  of  which  require¬ 
ments  are  to  be  met  through  training.  Decisions  about  the  functions  of 
the  training  are  to  be  made  on  the  basis  of  costs  and  effects  to  the  opera¬ 
tional  system,  costs  to  the  training  subsystem,  and  constraints  of  time, 
resources,  and  feasibility.  w 

•  Systems  analysis  involves  the  empirical  and  iterative  evaluation  of- alterna¬ 
tives.  The  impact  of  alternative  training  configurations  on  the  «fcrational 
system  often  cannot  be  clearly  anticipated.  Also,  principles  fofynalyzing 
training  requirements  and  decision  rules  for  selecting  and  organizing  instruc¬ 
tional  content  to  meet  learning  goals  are  not  well  developed.  Training  devel¬ 
opment  requires  a  series  of  approximations  involving  test  and  modification. 

•  Evaluation  criteria  are  specified.  Within  the  training  subsystem,  instructional 
development  is  evaluated  in  terms  of  specific  criteria  that  J*ve  previously 
been  specified  (i.e.,  behavioral  objectives  stated  in  terms  of  tasks,  standards, 
and  conditions)  and  the  extent  to  which  costs  have  been  minimized  in 
meeting  these  criteria  (e.g.,  lean  instruction).  Evaluation  of  the  adequacy  of 
the  ISD  process  itself  (in  contrast  to  the  training  it  has  generated)  must  be 
accomplished  in  terms  of  criteria  external  to  the  training  subsystem.  This 
poses  a  special  problem  in  that  criterion  measures  of  the  effectiveness  of 
the  operational  system  are  generally  not  available. 

While  ISD  interfaces  with  other  components  of  the  operational  system,  its  focus  is 
clearly  upon  the  training  subsystem.  For  example,  although  theoretically  it  would  be 
possible,  the  process  does  not  begin  with  complete  freedom  in  conceptualizing  the  role 
of  the  human  being  in  the  operational  system,  that  is,  allocating  functions  to  be  per¬ 
formed  by  man  and  functions  to  be  performed  by  hardware.1  Rather,  ISD  begins  with 
either  the  identification  of  a  need  to  develop  training  for  a  new  job  or  the  recognition 
of  a  problem  in  existing  training  for  an  old  job.  In  either  case,  job  requirements  are 
largely  fixed  by  the  time  ISD  and  the  training  developer  enter  the  picture.  As  we  shall 
later  see,  the  first  ISD  decision  that  has  an  impact  on  the  operational  system  is  one  in 
which  tasks  are  selected  for  training;  that  is,  of  all  the  tasks  that  must  be  performed  by 
a  human  being,  those  for  which  training  will  be  provided  are  specified. 

The  emergence  and  codification  of  the  ISD  process  have  overlapped  with  the 
appearance  of  a  variety  of  techniques  that  are  currently  popular  in  contemporary  train¬ 
ing  technology  (e.g.,  criterion-referenced  testing,  individualization  and  self-pacing  of 
instruction,  audiovisual  media,  computer -assisted  instruction,  computer-managed  instruc¬ 
tion).  As  a  consequence,  there  is  sometimes  a  tendency  for  such  techniques  to  be 
considered  virtually  synonymous  with  ISD.  Such  confusion  works  to  the  detriment  of 
ISD.  ISD,  however,  has  its  roots  in  systems  analysis,  and  as  such  is  primarily  a  process 
through  which  training  alternatives  are  selected.  It  should  not  be  identified  solely  with 
the  application  of  specific  training  strategies  or  products. 

*The  Interservice  Procedures  for  ISD  (Executive  Summary)  explicitly  states  that  ISD  follows  a 
specification  of  requirements  of  the  weapon,  force  structure,  strategy,  and  personnel  subsystems.  AF 
Manual  50-2  states  that  ISD  “  .  .  .  identifies  the  nature  and  scope  of  the  human  role  in  the  system  ...” 
but  no  further  attention  is  given  to  the  concept.  AF  Pamphlet  50-58,  which  provides  detailed  procedures 
for  implementing  the  concepts  of  AF  Manual  50-2,  omits  this  phase. 
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SUMMARY  MODEL  OF  ISD 

A  brief  description  of  the  components  or  steps  of  a  generalized  ISD  model  follows. 

It  is  based  on  an  analysis  of  the  major  models  now  in  use  in  the  Services.'  A  later 
description  discusses  alternative  placement  of  some  of  the  steps. 

Components  of  the  model  are  necessarily  depicted  in  a  linear  sequence.  In  practice, 
some  of  the  steps  may  be  conducted  concurrently  while  others  must  be  done  sequentially. 
In  keeping  with  a  systems  approach,  all  models  emphasize  the  interrelationships  of  the 
steps.  Adjustments  are  to  be  made  during  successive  iterations  of  the  process. 


Need/Discrepancy  Analysis 

The  first  step  in  ISD  consists  of  actions  that  either  reveal  or  confirm  a  need  to  develop 
or  revise  training.  If  a  need  to  revise  training  is  identified,  the  point  of  entry  into  ISD 
and  the  boundaries  of  the  redesign  process  are  indicated. 


Identification  of  Job  Requirements 

Job  requirements  are  identified  in  what  is  clearly  the  keystone  of  the  ISD  process. 
This  step  consists  of  (a)  developing  a  list  of  tasks  encountered  in  the  job,  (b)  verifying  the 
list  with  a  representative  sample  of  job  incumbents,  and  (c)  gathering  task  priority  informa¬ 
tion  (e.g.,  frequency  and  difficulty  of  task  performance). 


Selection  of  Tasks  for  Training 

Tasks  for  which  training  will  be  given  are  selected  from  those  identified  earlier  during 
job  analysis.  Decisions  are  based  on  task  priority  information,  training  resources,  and  costs. 


Analysis  of  Tasks 

Conditions  of  performance,  behavioral  elements,  and  standards  of  performance 
within  the  job  environment  are  described  for  tasks  that  have  been  selected  for  training. 
Task  analysis  provides  information  needed  later  for  developing  both  Job  Performance 
Measures  and  training  objectives. 


Construction  of  Job  Performance  Measures 

Job-referenced  tests  of  performance  are  constructed  to  serve  as  criteria  for  assessing 
the  capabilities  of  trainees  to  meet  job  requirements. 


1  Several  steps  included  in  the  Service  models  that  do  not  directly  affect  the  design  process 
(Implement  Instructional  Plan,  Conduct  Instruction,  Revise  System)  have  been  omitted  from  this 
generalized  ISD  model  and  from  later  d'scussion  of  the  individual  models. 
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Selection  of  Setting 

The  locus  of  instruction  for  tasks  selected  for  training  is  identified.  Decisions  are 
based  on  such  factors  as  cost,  characteristics  of  instructional  content,  student  load, 
and  special  constraints  (e.g.,  range  requirements).  Decisions  about  on-the-job  training 
options  must  consider  their  impact  on  the  force  structure. 


Development  of  Training 
Objectives  and  Objectives  Hierarchies 

Most  of  the  preceding  steps  have  dealt  with  job  requirements  and  job-related  decisions. 
This  step  represents  a  shift  in  focus  and  constitutes  the  first  analytic  step  in  the  develop¬ 
ment  of  instruction.  Objectives  are  statements  of  performance  requirements,  conditions, 
and  standards.  Terminal  objectives  specify  the  overall  behavior/outcome  of  task  perform¬ 
ance.  Intermediate  objectives  specify  the  behavioral  elements  and  the  basic  skills  and 
knowledge  that  mediate  terminal  objectives.  Intermediate  objectives  are  identified  by 
constructing  objectives  hierarchies,  that  is,  multilevel  displays  of  a  successive  breakdown 
of  terminal  objectives  into  their  supporting  behavior  and  mediating  skill  and  knowledge. 


Development  of  Achievement  Tests 

Achievement  tests  are  referenced  to  instructional  objectives.  Their  form  depends  on 
the  nature  of  an  objective;  thus,  paper-and-pencil  tests  are  used  to  assess  knowledge,  per¬ 
formance  tests  are  used  to  assess  behavior,  and  so  on.  Where  objectives  are  similar  to 
job  requirements,  achievement  tests  may  be  similar  or  identical  to  the  job  performance 
measures  developed  earlier. 


Identification  of  Entry  Behavior 

Instructional  objectives  developed  earlier  were  based  on  estimates  of  student  capa¬ 
bilities  at  the  time  of  entry  into  training.  These  estimates  are  now  verified  to  determine 
whether  trainees  possess  these  previously  presumed  capabilities.  Objectives  are  modified 
to  make  them  consistent  with  the  capabilities  of  entering  trainees.  Entry  behavior  may 
also  be  assessed  later,  when  actual  instruction  has  begun,  to  determine  whether  capa¬ 
bilities  of  the  entry  population  have  changed  over  time,  and  to  provide  diagnostic  infor¬ 
mation  for  placement  purposes  when  instruction  is  individually  prescribed. 


Classification  of  Objectives  and 
Selection  of  Instructional  Activities 

Objectives  are  classified  as  to  type  of  knowledge  and  skill.  Instructional  events 
and  conditions  are  selected  in  accordance  with  the  characteristics  of  the  capability  to  be 
acquired  (e.g.,  intellectual  skill,  information,  motor  skill)  and  the  level  of  learning 
required  (e.g.,  familiarization,  application). 
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Selection  of  Instructional  Methods 


Instructional  methods  are  the  means  by  which  trainees  are  brought  into  contact  with 
instruction.  They  are  defined  in  terms  of  the  size  and  location  of  the  instructional  group, 
degree  of  individualization,  and  type  of  instructional  pacing,  tutoring,  and  management. 
Methods  are  selected  so  as  to  maximize  the  effectiveness  and  efficiency  of  instruction, 
and  selection  depends  upon  the  particular  characteristics  of  the  skills  and  knowledge  to 
be  acquired,  the  nature  of  the  trainee  population,  the  setting,  and  administrative  require¬ 
ments  and  constraints. 


Selection  of  Media 

Media  are  selected  depending  on  the  requirements  of  instructional  activities  and 
methods  previously  specified.  Media  selected  must  possess  stimulus  characteristics  and 
response  opportunities  to  support  the  instructional  activities  (e.g.,  texts  to  communicate 
verbal  information,  operational  equipment  or  simulators  to  provide  practice  in  tracking). 
Media  must  also  be  appropriate  to  the  instructional  methods  (e.g.,  lecture  or  film  for 
group  instruction,  sound/slide  for  self-paced  instruction).  From  media  that  meet  these 
requirements,  final  selection  is  made  on  the  basis  of  such  factors  as  cost,  availability, 
production  capabilities,  size  of  trainee  population,  and  training  setting. 


Grouping  and  Sequencing  of  Instruction 

Topics  are  usually  grouped  on  the  basis  of  commonality  of  subject  matter  and 
anticipated  transfer  of  learning.  Dependent  relationships  among  training  objectives 
primarily  determine  the  sequence  of  instruction.  Independent  and  coordinate  objectives 
are  sequenced  in  accordance  with  such  principles  as  simple  to  complex,  familiar  to 
unfamiliar  and  job  order. 


Development  of  Plan  for 
Authoring  and  Managing  Instruction 

A  written  plan(s)  is  prepared  to  transmit  the  decisions  and  specifications  determined 
in  previous  ISD  steps  to  those  who  will  author,  conduct,  and  manage  the  instructional 
program.  The  plan  includes  (a)  guidelines  and  supporting  information  for  the  preparation 
of  lesson  content  (e.g.,  how  stimulus  material  is  to  be  presented  and  performance 
elicited,  types  and  schedule  of  testing,  trainee  literacy  characteristics),  (b)  description  of 
student  management  procedures  and  instructor  roles,  and  (c)  administrative  information 
(e.g.,  class  size,  equipment  use,  and  facility  schedules). 


Review  and  Selection  of  Existing  Materials 

Before  new  instructional  materials  are  produced,  existing  materials  are  examined  to 
determine  whether  they  can  be  used  or  adapted  for  use.  In  order  to  be  adopted,  existing 
materials  must  conform  to  prior  specifications  for  instructional  activities,  methods,  and 
media. 
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Authoring  of  Instruction 


New  instruction,  intentionally  lean  in  nature,  is  developed  in  accordance  with  prior 
specifications.  As  first  drafts  are  prepared,  small  portions  are  tried  out  on  persons  repre¬ 
sentative  of  the  entry  population,  and  are  modified  and  augmented  as  needed. 


Validation  of  Instruction 

In  validation,  newly  developed  instruction  is  evaluated  prior  to  implementation, 
under  conditions  that  closely  approximate  its  intended  use.  Lessons  or  an  entire  course 
is  administered  to  small  groups  of  trainees;  objective-referenced  achievement  tests  are 
given  to  determine  the  effectiveness  of  instruction;  and  training  is  modified  as  necessary 
to  meet  standards. 


Internal  Evaluation 

Internal  evaluation  consists  of  continuing  assessment  of  the  effectiveness  of  instruc¬ 
tion  when  it  has  been  implemented.  The  principal  measure  of  effectiveness  is  the 
trainees’  performance  on  objective-referenced  achievement  tests  (product  evaluation).  To 
aid  in  identifying  any  deficiencies  that  may  be  revealed,  internal  evaluation  includes 
analysis  of  how  the  ISD  process  was  actually  carried  out  (process  evaluation). 


External  Evaluation 

External  evaluation  occurs  when  graduates  of  a  training  program  have  reached  the 
job.  Whereas  internal  evaluation  determines  whether  trainees  are  attaining  course  objec¬ 
tives,  external  evaluation  uses  an  external  standard,  job  performance,  to  evaluate  pro¬ 
ficiency  and  the  adequacy  of  the  instructional  process.  Information  is  usually  obtained 
by  administering  questionnaires  to  graduates  and  their  supervisors;  occasionally  it  is 
obtained  through  testing  or  observation  of  performance. 


ORIGIN  AND  PURPOSE  OF  THE  STUDY 

ISD  models  and  their  forerunners  have  been  in  use  for  approximately  ten  years'  — 
a  long  enough  period  so  that  a  study  of  how  adequate  they  are,  and  how  they  are  being 
used,  can  be  undertaken.  In  late  1977,  the  Office  of  Assistant  Secretary  of  Defense 
(MRA&L)  requested  HumRRO  to  undertake  such  a  study.  The  objectives  were  to 
gather  information  about  the  effectiveness  of  the  various  ISD  methodologies  and  to 
determine  how  they  were  being  applied  in  each  of  the  Services.  What  problems  had 
arisen  in  their  application?  How  might  the  procedures  be  made  more  effective? 

'Montemerlo  and  Tennyson,  loc  cit.,  indicate:  “The  Navy  first  entered  this  field  with  the  devel¬ 
opment  of  the  Training  Situation  Analysis  (TSA)  methodology  (Bertin,  1963;  VanAlbert,  et  al.,  1964; 
Chenzoff  &  Folley,  1965).  The  Army  began  with  Project  Minerva,  an  Army  Security  Agency  study 
which  resulted  in  the  Design  of  Instructional  Systems  (DIS)  manual  in  1966  .  .  .  Two  other  influen¬ 
tial  manuals  of  this  period  were  Butler’s  (1967)  Instructional  Systems  Development  (ISD)  manual, 
which  was  written  under  Job  Corps  auspices,  and  Rundquist’s  course  design  manual  (1966,  1967,  1970) 
which  was  developed  at  the  Navy  Personnel  Research  and  Development  Center.” 
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'  While  a  systems  approach  to  training  has  many  potential  advantages,  it  is  difficult, 
time  consuming,  and  costly  to  carry  out.  It  will  be  worth  the  cost  and  effort  only  to 
the  extent  that  it  is  being  implemented  so  as  to  realize  its  potential.  For  example,  con¬ 
siderable  data  and  information  from  a  variety  of  sources  both  within  and  without  the 
training  subsystem  must  be  developed  and  analyzed.  Interdependencies  in  the  develop¬ 
ment  process  make  coordination  necessary  among  training  developers  concerned  with 
different  phases.  In  each  phase  there  must  be  accomodation  to  information  and  products 
developed  in  prior  phases.  The  requirement  for  continuous  verification  and  revision, 
while  a  major  strength  of  the  systems  approach,  is  particularly  demanding.  To  achieve 
the  system’s  goals,  instructional  systems  developers  must  be  able  to  maintain  sustained 
commitment  to  a  repetitive  and  often  difficult  process. 

At  least  one  earlier  study  revealed  many  problems  when  there  was  an  attempt  to 
institutionalize  a  systems  approach  to  training  development.  Ricketson,  et  al.,  interviewed 
personnel  who  were  attempting  to  implement  USCONARC  Regulation  350-100-1,  Systems 
Engineering  of  Training  -  Course  Design,  at  the  U.S.  Army  Aviation  School.1  Even 
though  the  procedures  in  the  CON  ARC  Regulation  were  far  less  demanding  than  those 
called  for  in  current  ISD  models,  significant  problems  were  encountered  with  their  inter¬ 
pretation  and  application.  Guidance  was  found  to  be  incomplete,  contradictory,  and 
lacking  in  detail;  individual  steps  in  the  process  were  difficult  to  relate  to  one  another; 
job  analysis  data  collected  in  the  field  omitted  important  tasks  and  described  others  in 
excessive  detail;  product  review  was  slow  and  contributed  to  wasteful  false  starts;  per¬ 
sonnel  turnover  was  disruptive;  and  so  forth. 

The  magnitude  of  the  requirements  make  it  reasonable  to  ask  how  ISD  is  currently 
working  in  actual  practice.  Are  the  methodologies  adequate?  Are  they  being  complied 
with?  Do  they  make  a  difference? 


SCOPE  OF  THE  STUDY 

As  has  been  indicated,  the  absence  of  a  criterion  of  system  effectiveness  precludes 
evaluating  the  effects  of  ISD  on  the  basis  of  total  system  performance.  Instead,  the 
study  focuses  on  the  training  subsystem  and  within  this  subsystem  is  restricted  to  an 
analysis  of  methodology,  process,  and  intermediate  products.  The  job  performance  of 
course  graduates  has  not  been  examined. 

The  analysis  of  ISD  in  this  study  is  based  upon  information  developed  in  three 
major  activities: 

•  Analysis  of  the  primary  documents  currently  in  use  in  the  Army,  Navy,  Marine 
Corps,  and  Air  Force  for  guidance  in  conducting  ISD. 

•  Questionnaire  survey  of  209  units,  agencies,  and  schools  where  instruction  is 
developed  in  the  Services. 

•  Detailed  interview  of  training  developers  from  33  of  the  organizations  surveyed 
to  determine  how  57  specimen  courses  were  designed. 

Chapter  2  of  the  report  contains  an  analysis  of  the  ISD  process  and  of  the  major  ISD 
models  being  used  in  the  Services.  Chapter  3  describes  the  questionnaire  survey  undertaken 
to  select  a  sample  of  ISD  applications  for  detailed  analysis.  Chapter  4  describes  the  role  and 
structure  of  organizations  performing  ISD  in  each  of  the  Services.  Chapter  5  describes  how 
the  models  are  being  applied  as  revealed  in  the  interviews  with  developers  of  training.  Chap¬ 
ter  6  summarizes  the  findings  of  the  study  and  makes  recommendations. 


1  Ricketson,  Darwin  S.,  Schulz,  Russel  E.,  and  Wright,  Robert  H.  Review  of  the  CONARC  Systems 
Engineering  of  Training  Program  and  Its  Implementation  at  the  US  Army  Aviation  School,  Consulting 
Report,  HumRRO,  April  1970. 
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Chapter  2 

ANALYSIS  OF  ISD  PROCESS  AND  MAJOR  ISD  MODELS 


INTRODUCTION 

It  is  appropriate  to  begin  an  analysis  of  Instructional  System  Development  activities 
in  the  Armed  Services  with  an  examination  both  of  the  process  itself  and  of  the  models 
and  manuals  that  are  intended  as  procedural  guides  to  the  process.  The  issue  of  whether 
ISD  can  or  cannot  be  reduced  to  procedures  to  be  applied  by  nonprofessional  practitioners 
has  been  discussed  elsewhere.1 2 * *  Continuing  the  debate  is  not  likely  to  be  fruitful.  In  any 
event  the  majority  of  persons  who  are  cunently  attempting  to  apply  ISD  in  the  Armed 
Services  have  acquired  their  understanding  of  the  process  primarily  from  descriptive  manuals 
or  from  instruction  derived  from  the  manuals.  It  cannot  be  expected  that  ISD  applications 
will  be  any  better  than  the  guidance  they  are  based  upon. 

Primarily  ISD  guidance  documents5  currently  in  use  in  the  Services  are: 

•  Interservice  Procedures  for  Instructional  Systems  Development  (1975),  pub¬ 
lished  by  the  Army  as  TRADOC  Pamphlet  350-30,  and  by  the  Navy  as 
NAVEDTRA  106 A— Developed  for  the  Army  and  later  approved  by  the  Inter¬ 
service  Committee  on  Instructional  Systems  Development.  Contains  detailed 
procedures  for  conducting  ISD.  Hereinafter  referred  to  as  the  ITRO  model. 

•  Marine  Corps  Order  P1510.23B  (1978)— A  greatly  reduced  version  of  the  ITRO 
model  incorporating  documentation  requirements  specific  to  the  Marine  Corps. 

•  Air  Force  Manual  50-2,  Instructional  Systems  Development  (1970,  1975)— An 
overview  of  the  ISD  process. 

•  Air  Force  Pamphlet  50-58,  Handbook  for  Designers  of  Instructional  Systems 
(1973,  1974)— Detailed  procedures  for  implementing  the  concepts  in  AF 
Manual  50-2. 


1  Montemerlo  and  Tennyson,  op.  cit.,  pp.  10-12. 

2  In  addition  to  these  primary  documents,  various  modifications  and  supplementary  documents  are 

in  use.  For  example,  the  U.S.  Army  Armor  School  has  published  the  Training  Development  Handbook 
(1977),  which  is  largely  a  revision  of  the  ITRO  model  that  also  incorporates  terminology  and  documenta¬ 
tion  requirements  specific  to  the  Army;  the  Chief  of  Naval  Technical  Training  has  published  CNTT-A10 
Procedures  for  the  Planning,  Design,  Development,  and  Management  of  Navy  Technical  Training  Courses 
(1976),  specifying  procedures  and  documentation  formats  to  be  used  in  conjunction  with  the  ITRO 
model;  the  U.S.  Naval  Health  Sciences  and  Education  Command  has  published  the  HSETC  Handbook 

for  Implementation  (1977),  intended  to  provide  supplementary  guidance  for  use  of  the  ITRO  model 
within  the  command;  the  Chief  of  Naval  Education  and  Training  has  published  NAVEDTRA  106A 
Supplement  No.  2,  Instructional  Program  Development,  Training  Task  Analysis  Procedures  (1978)  and 
NAVEDTRA  110,  Procedures  for  Instructional  Systems  Development  (1978),  providing  Navy-specific 

guidance  for  implementing  the  ITRO  model;  the  Naval  Training  Equipment  Center  has  sponsored  devel¬ 
opment  of  Military  Specification  MIL-T-29053,  Training  Requirements  for  Aviation  Weapon  Systems 
(1977),  to  guide  ISD  efforts  by  civilian  contractors.  These  are  only  examples  of  the  various  supporting 
documents  that  have  been  published. 
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With  the  exception  of  Marine  Corps  Order  P1510.23B,  these  documents  generally 
are  available  in  each  of  the  Services.  One  should  not  assume,  for  example,  that  the  ITRO 
model  is  used  exclusively  in  the  Army  and  Navy  or  that  Air  Force  documents  are  never 
referred  to  outside  the  Air  Force. 

The  remainder  of  this  chapter  contains  the  step-by-step  analysis  of  the  ISD  process 
and  evaluation  of  the  three  major  manuals/models:  the  ITRO  model.  Marine  Corps 
Order  P1510.23B,  and  AF  Pamphlet  50-58.  The  steps  and  requirements  of  a  generalized 
ISD  model  have  been  derived  in  part  from  synthesizing  the  Service  models  and  in  part 
through  rational  analysis  of  their  intended  functions.  Several  steps  included  in  the  Service 
models  that  do  not  directly  affect  the  design  process  (Implement  Instructional  Plan,  Con¬ 
duct  Instruction,  Revise  System)  have  been  omitted  from  the  generalized  ISD  model. 

Each  of  the  following  sections  contains  a  description  of  a  specific  ISD  step;  a  summary 
of  prerequisite  conditions  and  information,  and  procedural  requirements  for  performing 
the  step;  an  analysis  of  the  instructions  provided  in  each  of  the  three  Service  models  for 
meeting  the  requirements;  and  a  summary  commentary. 


STEPS  OF  PROCESS  AND  MODELS 

Need/Discrepancy  Analysis 

ISD  originates  with  the  identification  of  a  need  to  develop  or  revise  instruction. 
Instruction  will  have  to  be  developed  if  human  performance  in  a  new  job  or  weapon  sys¬ 
tem  needs  to  be  trained;  instruction  will  have  to  be  revised  if  existing  training  is  not  suffi¬ 
ciently  effective  and  economical.  Some  models  explicitly  identify  an  analysis  of  needs 
as  part  of  ISD,  while  others  describe  it  as  an  antecedent  condition.  Whether  this  step  is 
viewed  as  lying  within  or  without  the  process  is  largely  a  matter  of  definition.  It  is  impor¬ 
tant,  however,  that  ISD  follow  a  determination  of  need  rather  than  be  undertaken  on  a 
priori  grounds.  The  process  itself  is  costly  and  should  be  justified. 

Prerequisite  conditions  for  determining  need  are  the  existence  and  maintenance  of 
procedures  for  detecting  indications  of  potential  discrepancies.  In  an  ideal  world,  dis¬ 
crepancies  in  training  would  be  identified  by  constantly  monitoring  (a)  job  requirements 
(does  job  analysis  indicate  that  job  demands  are  not  adequately  reflected  in  training  or 
that  they  have  changed  since  an  earlier  determination?),  (b)  job  performance  (does 
measurement  of  job  performance  indicate  that  job  incumbents  are  not  adequately  trained?), 
and  (c)  course  materials  (does  empirical  comparison  of  alternate  types  of  content  and 
media  indicate  that  training  is  not  maximally  cost  effective?).  These  ways  of  identifying 
discrepancies  are,  of  course,  extremely  costly.  In  the  real  world,  these  methods  must  be 
used  selectively,  so  there  should  be  a  means  to  determine  when  their  employment  is  appro¬ 
priate.  Continuous  job  analysis,  performance  testing,  and  empirical  comparisons  of  alter¬ 
nate  forms  of  training  are  neither  feasible  nor  desirable. 

Indications  of  changing  job  requirements  and  inadequate  training  may  be  as  diverse 
as  high  accident  rates,  reports  of  inadequate  performance  from  supervisors  and  field  com¬ 
manders,  and  discrepancies  between  course  syllabi  and  duties  reported  in  occupational 
surveys.  Symptoms  that  training  may  not  be  effective  and  efficient  are  to  be  seen  in 
such  indicators  as  high  course  attrition  rates,  unusual  course  length  and  cost,  or  negative 
comments  from  students  and  instructors. 

When  a  potential  discrepancy  is  identified,  additional  information  is  gathered  as  needed 
to  confirm  and  define  the  problem  and  to  suggest  the  appropriate  locus  for  corrective 
action,  that  is,  where  the  ISD  process  should  be  entered  for  remediation.  If,  for  example, 
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there  is  an  apparent  discrepancy  between  job  requirements  and  the  performance  of  job 
incumbents,  existing  training  will  be  examined  to  determine  whether  the  problem  lies  in 
an  original  analysis  of  the  job,  whether  errors  have  occurred  in  the  translation  of  job 
requirements  into  objectives  of  training,  whether  the  way  in  which  training  is  actually 
conducted  is  ineffective,  and  so  on.  The  extent  of  ISD  that  subsequently  will  be  under¬ 
taken  depends  both  upon  the  origin  of  the  problem  and  upon  its  specific  characteristics. 
If  analysis  of  the  problem  reveals  a  need  to  add  a  few  new  objectives  requiring  training 
similar  to  that  already  given,  it  is  not  likely  that  the  entire  ISD  process  will  be  necessary, 


Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Information  is  available  about  possible  discrepancies  between  training  and 
field  requirements. 

Procedures 

1.  Analyze  initial  and  supplementary  information. 

2.  Identify  and  specify  discrepancy. 

3.  Specify  ISD  entry  point  and  boundaries  of  redesign  process. 

Analysis  of  Service  Models 

ITRO  Model.  The  Executive  Summary  presents  a  diagram  to  show  that  ISD  is  pre¬ 
ceded  by  an  analysis  of  weapons  systems,  the  force  structure,  manpower,  and  existing 
strategies.  This  analysis,  the  diagram  indicates,  leads  to  the  identification  of  any  “perform¬ 
ance  discrepancy”  between  requirements  on  the  one  hand,  and  existing  programs  on  the 
other.  If  no  discrepancy  exists,  of  course,  nothing  need  be  done.  If  there  is  a  discrepancy, 
ISD  will  provide  “alternative  solutions  for  selected  defined  performance  discrepancies.” 

The  text  of  the  Executive  Summary  indicates  that  discrepancies  arise  from  such  happen¬ 
ings  as  changes  in  technology  or  force  structure.  It  suggests  an  equivalence  between  this 
type  of  analysis  and  the  Phase  I  activities  of  the  ITRO  model  (job  analysis,  selection  of 
setting),  but  does  not  make  this  explicit. 

In  Step  1.1  (Analyze  Job),  the  ITRO  model  states  that  the  first  step  of  ISD  is  to 
identify  the  discrepancy  between  existing  and  required  training,  and  that  the  first  require¬ 
ment  in  this  process  is  an  adequate  job  analysis.  If  no  job  analysis  has  previously  been 
accomplished,  the  model  specifies  that  the  ISD  process  should  be  undertaken  from  the 
beginning.  However,  if  a  training  program  already  exists  that  is  based  on  an  adequate 
analysis,  and  if  tasks  had  been  selected  and  Job  Performance  Measures  constructed  in 
accordance  with  ITRO  procedures,  the  model  does  not  specify  further  how  to  identify  the 
discrepancy. 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  model  begins  with  job  analysis,  and 
does  not  explicitly  identify  a  need /discrepancy  analysis  as  part  of  ISD. 

Air  Force  Pamphlet  50-58.  The  AFP  50-58  model  does  not  contain  procedures  for 
a  need  or  discrepancy  analysis,  but  does  refer  to  the  Air  Force  Manual  50-2  ISD  model. 

In  that  model,  "Analyze  System  Requirements”  is  the  first  of  five  steps.  This  includes 
an  analysis  focusing  on  the  human  role  in  the  total  operational  system,  eventually  reaching 
the  level  of  specific  job  requirements.  The  ISD  procedures  outlined  in  AFP  50-58,  then, 
would  presumedly  be  initiated  when  such  a  prior  analysis  indicated  a  need  to  develop  or 
revise  training. 
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Commentary.  None  of  the  models  provides  specific  guidance  for  identifying  dis¬ 
crepancies,  other  than  between  the  need  for  training  and  the  lack  of  a  training  program. 
Although  the  ITRO  model  states  that  procedures  will  be  given  to  assist  the  developer  in 
“determining  the  exact  location  of  the  discrepancy  in  the  training  program,”  the  only 
procedure  actually  given  for  existing  training  programs  is  to  undertake  ISD,  and  the  only 
point  recommended  for  entering  the  ISD  process  is  at  the  beginning.  No  provision  is 
made  for  any  alternative  means  of  identifying  and  resolving  performance  discrepancies. 


Identification  of  Job  Requirements 

No  decisions  about  training  can  be  made  until  an  accurate  picture  of  the  requirements 
of  the  job  is  obtained.  Job  analysis  and  task  analysis,  therefore,  are  first  steps  in  develop¬ 
ing  training.  Job  analysis  determines  what  tasks  are  performed  on  the  job  and  provides 
information  that  can  be  used  to  establish  instructional  priorities.  Task  analysis  investi¬ 
gates  the  conditions,  operations,  and  standards  that  define  job  performance.  Because  job 
analysis  and  task  analysis  involve  different  activities,  they  are  ordinarily  performed  as 
separate  steps  of  ISD.  Task  analysis  generally  requires  a  lengthy  and  detailed  specification 
of  the  task’s  behavioral  elements.  It  is  probably  not  efficient  to  undertake  it  before  elimi¬ 
nating  some  tasks  in  the  step,  Selection  of  Tasks  for  Training.1 

Job  analysis  consists  of  three  parts:  compiling  a  provisional  list  of  tasks  believed  to 
comprise  the  job,  verifying  the  accuracy  of  the  list  and  adjusting  it  as  necessary,  and 
gathering  task  priority  information.  Task  lists  are  developed  in  two  phases  to  increase 
their  reliability.  Since  ISD  is  a  process  for  developing  training,  it  is  reasonable  to  expect 
that  trainers  will  be  responsible  for  the  analysis,  but  their  familiarity  with  field  require¬ 
ments  can  vary  widely.  A  verification  phase  permits  the  provisional  task  inventory  to  be 
administered  to  a  larger  sample  of  operational  personnel  for  validation. 

The  initial  task  list  can  be  generated  in  various  ways,  ranging  from  recall  by  a  single 
subject  matter  expert  to  extensive  observation  of  job  incumbents.  The  accuracy  of  the 
final  job  description  is  likely  to  be  influenced  by  the  comprehensiveness  of  the  initial 
inventory.  Though  the  verification  phase  allows  tasks  to  be  added  to  the  list,  there  is 
no  guarantee  that  respondents  will  do  so. 

Verification  may  also  take  different  forms  ranging  from  the  use  of  a  second  group  of 
subject  matter  experts  who  are  convened  to  review  the  original  task  list,  to  a  phone  or 
much  broader  mail  survey.  In  concept,  verification  should  provide  for  validation  with  a 
representative  sample  of  job  incumbents.  Depending  on  the  job,  this  usually  implies  a 
larger  and  more  varied  sample  than  is  provided  in  a  second  group  of  subject  matter 
experts. 

Task  priority  information  is  data  about  such  factors  as  billets  where  the  task  is 
performed,  percent  of  persons  performing,  frequency  of  performance,  difficulty  in  learn¬ 
ing  and  performing,  probability  of  deficient  performance,  consequences  of  deficient 
performance,  and  average  time  between  training  and  performance.  These  inevitably  are 
subjective  estimates.  To  increase  their  reliability,  this  information  should  usually  be 
gathered  from  a  fairly  large  sample  of  job  incumbents.  Where  task  lists  are  being  verified 
by  mail  or  some  other  form  of  large-scale  survey,  task  priority  information  may  be 
gathered  at  the  same  time.  In  this  case,  however,  a  second  survey  will  be  needed  to 
obtain  information  about  tasks  added  to  the  list  during  verification. 


1  See  Analysis  of  Tasks,  p.  17,  for  a  discussion  of  the  location  of  this  step  in  the  ISD  process 


13 


Summary  of  Requirements 


Prerequisite  Conditions  and  Information 

None 

Procedures 

1.  Construct  provisional  task  list. 

2.  Verify  and  revise  on  basis  of  review/survey  of  job  incumbents. 

3.  Collect  task  priority  information. 

Analysis  of  Service  Models 

1TRO  Model.  On-site  interviews  (which  can  include  observation)  are  recommended 
for  constructing  the  initial  task  list.  Methods  mentioned  as  less  effective  include  the 
questionnaire  survey,  jury  of  experts,  and  group  interview.  For  verifying  the  accuracy 
of  the  task  list,  the  ITRO  model  specifies  the  questionnaire  survey  method.  When  a 
current,  complete  task  list  already  exists,  the  initial  data-gathering  phase  is  omitted,  and 
only  the  verification  phase  occurs. 

For  collecting  task  priority  information,  the  model  recommends  sending  a  question¬ 
naire  to  incumbents  and  supervisors,  either  at  the  same  time  as  task  list  validation,  or 
in  the  next  ISD  step. 

Marine  Corps  Order  P1510.23B.  Several  methods  are  listed  for  collecting  job  data, 
such  as  interviews,  observation,  analyzing  the  content  of  training  programs,  or  assump¬ 
tions.  The  manual  states  that  each  method  has  advantages  and  disadvantages  relative  to 
the  others,  but  does  not  tell  what  these  are,  nor  recommend  any  method  over  another. 

(A  single  exception  is  the  statement  that  “detailed  personal  interview”  is  superior  to 
“assumption  analysis.”  This  is  presented  as  an  example  of  why  a  record  should  be 
retained  of  what  methodology  was  employed.)  To  verify  the  accuracy  of  the  task  list, 
the  Marine  Corps  model  specifies  that  it  be  “double  checked”  by  a  group  of  subject 
matter  specialists,  including  supervisory  and  instructor  personnel.  The  use  of  question¬ 
naires  to  collect  task  priority  data  is  not  discussed;  the  only  suggested  source  of  this 
information  is  Commandant  of  the  Marine  Corps  task  analyses. 

Air  Force  Pamphlet  50-58.  The  AFP  50-58  model  recommends  obtaining  a  verified 
task  list  from  an  Occupational  Survey,  if  available.  If  not,  the  model  recommends  using 
a  team  of  seven  or  eight  subject  matter  experts  to  develop  the  list,  based  on  such  factors 
as  specialty  descriptions,  familiarity  with  similar  jobs,  and  engineering  data.  Although 
implying  that  this  method  is  not  as  desirable  as  an  “in-depth  survey,”  AFP  50-58  notes 
that  the  time  and  resources  required  for  the  latter  are  seldom  available  to  training  devel¬ 
opers.  Interview,  observation,  and  questionnaire  are  also  mentioned  as  possible  data 
collection  methods.  To  verify  the  task  list,  suggested  methods  include  questionnaire, 
interview,  observation,  simulation,  and  assumption.  For  collecting  task  priority  infor¬ 
mation,  the  model  suggests  the  use  of  Occupational  Survey  Reports,  interviews  with 
subject  matter  experts,  and  questionnaires. 

Commentary.  While  the  ITRO  and  AFP  50-58  models  recommend  surveys  for 
collecting  data  about  job  tasks,  both  acknowledge  that  less  reliable  and  less  costly  methods 
may  have  to  be  used.  The  Marine  Corps  model,  with  the  exception  noted,  does  not 
identify  any  one  method  as  superior  for  compiling  and  verifying  task  lists,  and  contains 
almost  no  guidance  for  collecting  task  priority  information.  All  the  models,  then,  allow 
much  latitude  in  choice  of  methods.  Under  these  conditions,  the  accuracy  of  the  task 
list  and  the  reliability  of  task  priority  data  would  appear  to  depend  on  the  experience 
and  conscientiousness  of  the  persons  collecting  and  contributing  information,  rather  than 
on  the  particular  model  followed. 
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Selection  of  Tasks  for  Training 

Tasks  identified  during  job  analysis  are  classified  on  the  basis  of  whether  training 
will  or  will  not  be  provided.  Those  selected  will  become  the  basis  of  the  training  program, 
those  rejected  will  not  be  reconsidered  until  the  external  evaluation.  This  step  is  based 
on  the  premises  that  (a)  for  some  tasks  identified  by  job  analysis,  it  is  not  necessary  to 
provide  training,  and  (b)  resources  may  not  be  sufficient  to  provide  training  for  all  tasks 
on  which  training  would  be  desirable. 

Prerequisites  to  the  selection  process  are  a  list  of  tasks  that  comprise  the  job, 
information  about  the  tasks  for  establishing  their  importance  and  need  for  training,  and 
decision  rules  for  determining  training  priorities.  To  promote  objectivity,  the  information, 
and  the  procedures  for  processing  it,  should  be  developed  prior  to  the  selection  process. 

In  the  absence  of  a  parent  list  of  tasks,  selection  is  not  selection  at  all,  but  simply  a 
decision.  If  task  lists,  priority  information,  and  rules  of  application  are  developed  con¬ 
jointly  with  task  selection,  inevitably  the  latter  will  influence  the  former.  It  will  be 
difficult  to  avoid  establishing  priority  criteria  consonant  with  one’s  viewpoint  on  what 
is  important  to  train;  that  is,  one  easily  acquires  a  tendency  to  select  criteria  to  fit  tasks 
rather  than  selecting  tasks  on  the  basis  of  criteria.  This  is  especially  likely  because  the 
choice  of  criteria  itself  is  largely  based  on  judgment;  information  about  the  relation  of 
task  selection  criteria  to  total  system  performance  is  not  available.  At  present,  there  is 
no  way  to  test  the  adequacy  of  tasks  selected  in  terms  of  overall  system  effectiveness. 

Some  task  characteristics  that  are  used  to  determine  training  priorities  are  number 
and  percent  of  persons  performing,  frequency  of  performance,  learning  difficulty,  time 
spent  in  performance,  learning  decay  rate,  length  of  time  after  training  before  perform¬ 
ance,  and  criticality  (probable  consequences  of  inadequate  performance).  Characteristics 
such  as  frequency  of  performance  can  be  determined  empirically;  others,  such  as  cri¬ 
ticality,  by  consensus. 

After  identifying  which  tasks  are  desirable  to  train,  and  which  of  these  are  most 
important,  the  requirements  for  training  resources  are  considered.  The  final  selection  of 
tasks  to  be  trained  is  made  on  the  basis  of  training  resource  requirements  and  availability— 
that  is,  cost.  Some  factors  which  affect  the  number  of  tasks  for  which  training  can  be 
provided  are  the  number  of  graduates  required  and  the  demands  training  will  make  on 
time,  facilities,  equipment,  fuel,  and  instructional  personnel.  If  not  all  desirable  tasks 
can  be  trained,  less  important  tasks  are  rejected. 

A  factor  that  greatly  influences  the  demands  made  on  training  resources  is  the 
training  setting  (i.e.,  resident  instruction,  formal  on-the-job  training,  self-taught  packages, 
etc.),  although  to  avoid  possible  bias  in  task  selection,  the  setting  is  selected  in  a  sepa¬ 
rate  ISD  step.  If  the  setting  is  identified  before  the  tasks  are  selected,  there  may  be  a 
tendency  to  select  tasks  that  are  easy  or  economical  to  train  in  that  setting  and  to  avoid 
those  that  are  difficult  and/or  expensive.  Also  there  may  be  a  tendency  to  select  tasks 
that  have  been  taught  in  the  prescribed  setting  before. 

Although  the  desirability  of  isolating  task  selection  from  setting  selection  is  clear, 
there  is  a  contradiction  inherent  in  separating  the  two.  Tasks  are  ranked  in  importance 
on  the  basis  of  their  characteristics  but  selected  for  training  on  the  basis  of  available 
resources/cost,  and  cost  is,  in  turn,  greatly  conditioned  by  setting.  To  completely 
separate  these  processes  would  be  to  defer  consideration  of  perhaps  the  greatest  cost 
factor  until  after  tasks  have  been  selected,  when  this  selection  is  ultimately  constrained 
by  cost.  To  maintain,  insofar  as  possible,  the  advantages  of  separation,  and  yet  avoid 
this  error,  the  selection  of  tasks  and  setting  must  be  accomplished  in  a  series  of 
interactive  iterations. 
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Summary  of  Requirements 

Prerequisite  Conditions  and  Information. 

1.  Tasks  performed  in  the  job  are  listed. 

2.  Information  is  available  for  establishing  importance  of  tasks  and  need 
for  training. 

3.  Decision  rules  to  be  applied  to  task  information  are  available. 

Procedures 

1.  Apply  decision  rules  to  information  for  each  task  to  determine 
training  priorities. 

2.  Select  tasks  for  training  on  basis  of  training  priorities  and  resource 
availability. 

Analysis  of  Service  Models 

ITRO  Model.  The  first  step  of  the  ITRO  procedure  is  to  define  the  criteria  by 
which  task  importance  will  be  measured.  Percent  performing,  time  between  job  entry 
and  task  performance,  and  learning  difficulty  are  suggested  as  the  most  appropriate 
criteria  for  determining  the  absolute  importance  of  tasks,  and  criticality  (consequences 
of  inadequate  performance)  as  more  applicable  to  combat  tasks  than  noncombat.  It 
is  further  suggested  that  the  number  of  criteria  selected  be  generally  limited  to  about 
four.  Beyond  such  suggestions,  no  guidance  is  given  for  selecting  criteria. 

When  the  criteria  have  been  decided  upon,  questionnaires  are  constructed,  then  used 
to  collect  the  task  priority  information  from  incumbents  and  supervisors.  (This  may  be 
done  simultaneously  with  job  analysis,  in  the  preceding  step.)  The  data  gathered  are 
then  examined  to  identify  tasks  that  do  not  warrant  training.  This  is  a  rather  gross 
determination,  designed  to  weed  out  tasks  that  are  obviously  not  important.  The  model 
states  that  if  there  is  doubt  about  a  task’s  importance,  it  should  probably  be  retained 
at  this  point. 

Next,  available  training  resources  are  estimated,  as  an  aid  in  determining  about  how 
many  of  the  remaining  tasks  can  be  trained.  Then  the  tasks  in  which  training  is  needed 
the  most  are  selected,  up  to  the  tentative  number  to  be  trained;  another  20%  or  so  of 
tasks  next  in  importance  are  also  identified,  and  rank-ordered,  to  be  available  for  selec¬ 
tion  when  resource  availability  is  known  precisely.  The  model  states  that  tasks  which 
rank  lower  on  the  task  selection  criteria  are  less  in  need  of  training,  but  does  not  suggest 
methods  for  weighing  the  relative  importance  of  the  several  criteria,  nor  for  considering 
how  the  different  criteria  might  be  related  to  total  system  effectiveness. 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  model  provides  guidance  similar 
to  that  of  the  ITRO,  but  more  briefly  stated.  It  presents  eight  task  selection  criteria 
but  does  not  differentiate  among  them  as  to  applicability.  As  noted  earlier,  no  pro¬ 
cedures  are  described  for  collecting  task  priority  inforamtion,  other  than  obtaining  a  task 
analysis  report,  if  available. 

Air  Force  Pamphlet  50-58.  Three  criteria  are  specified  for  use  in  selecting  tasks; 
number  performing,  percent  performing,  and  criticality.  The  model  also  provides  rules 
for  applying  the  criteria  (e.g.,  if  the  number  performing  a  task  were  51-100,  and  the 
percent  performing  below  50%,  the  task  would  be  recommended  for  training  if  its 
correct  performance  would  increase  job  effectiveness,  but  not  if  its  correct  performance 
would  merely  increase  efficiency).  AFP  50-58  stresses  that  the  rules  for  applying  the 
criteria  are  only  guidelines  to  making  an  “initial  decision”  about  task  selection,  which 
must  be  verified  by  some  other  means.  No  guidance  is  provided  regarding  any  conflict 
between  training  requirements  and  availability  of  training  resources. 
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Commentary.  All  the  models,  then,  suggest  criteria  by  which  to  select  tasks  for 
training,  but  only  one  provides  a  formula  for  their  application— and  that  only  condi¬ 
tionally,  and  for  only  three  criteria.  Moreover,  the  Marine  Corps  model’s  lack  of  guidance 
for  collecting  task  priority  information  could  result  in  the  practice  of  exploring  priorities 
simultaneously  with  selecting  tasks,  thereby  neutralizing  the  advantages  of  collecting 
priority  information  in  a  separate  step. 

To  specify  more  fully  how  to  select  and  weigh  various  criteria,  information  is 
clearly  needed  concerning  how  the  criteria  are  related  to  job  performance.  It  is  not 
known,  for  example,  how  total  system  effectiveness  is  changed  as  a  result  of  selecting 
difficult  tasks,  in  contrast  to  tasks  with  high  percentages  of  total  time  spent  performing, 
or  vice  versa.  Indeed,  for  almost  every  criterion  suggested  a  rationale  can  be  found  for 
using  it  to  reject  tasks  instead  of  selecting  them.  For  example,  it  can  be  argued  that 
training  should  be  given  for  more  frequently  performed  tasks  because  this  will  insure 
that  the  effects  of  training  will  be  more  widely  felt.  On  the  other  hand,  it  is  not  too 
extreme  to  suggest  that  training  not  be  given  for  the  most  frequently  performed  tasks 
on  the  premise  that  incumbents  will  have  a  better  opportunity  to  learn  them  on  the 
job.  In  the  absence  of  information  about  how  system  effectiveness  is  related  to  task 
selection  criteria,  rational  analysis  alone  cannot  be  used  to  select  the  criteria. 


Analysis  of  Tasks 

In  the  context  of  ISD,  task  analysis  is  a  description  of  when  and  how  within  the 
job  environment  the  performance  of  a  task  is  required.  Thus  it  consists  of  specifying 
conditions  of  performance  along  with  initiating  cues,  behavioral  elements,  and  standards 
of  performance.  The  information  generated  is  later  used  in  constructing  Job  Performance 
Measures  and  in  developing  objectives. 

The  only  prerequisite  for  task  analysis  is  that  tasks  to  be  analyzed  have  been  iden¬ 
tified.  It  therefore  can  be  undertaken  almost  concurrently  with,  and  is  considered  by 
some  a  part  of,  job  analysis.  Job  analysis,  however,  consists  essentially  of  a  rational 
reconstruction  of  job  requirements  followed  by  a  survey  of  job  incumbents.  Task  anal¬ 
ysis,  on  the  other  hand,  requires  a  detailed  specification  of  the  elements  of  task  perform¬ 
ance.  Because  different  activities  are  involved  in  the  two  functions  and  because  task 
analysis  is  often  a  lengthy  and  costly  process,  it  should  usually  be  undertaken  only  for 
those  tasks  that  have  previously  been  selected  for  training. 

Though  it  is  generally  more  efficient  to  do  task  analysis  after  task  selection,  this 
order  of  events  can  be  a  source  of  problems.  The  longer  that  task  analysis  is  delayed, 
the  greater  the  possibility  that  it  may  become  confounded  with  decisions  about  training. 
When  tasks  have  already  been  selected  for  training,  for  example,  there  may  be  a  greater 
tendency  to  describe  them  as  they  will  be  performed  in  training  rather  than  as  they  are 
performed  on  the  job.  It  is  important  to  emphasize  that  while  training  conditions  and 
standards  (established  later  in  the  ISD  process)  may  differ  considerably  from  those  of 
the  job,  it  is  not  the  purpose  of  task  analysis  to  determine  how  a  task  will  be  trained. 
Rather  it  is  to  discover  under  what  conditions  the  task  is  actually  performed,  and  what 
standards  of  performance  are  required  in  the  job.  Also,  when  selecting  a  task  for  train¬ 
ing  means  that  it  must  be  analyzed  and  rejecting  it  means  that  no  analysis  will  be 
required,  there  can  be  a  tendency  to  select  tasks  that  have  few  behavioral  elements,  or  on 
which  the  standards  are  easy  to  identify,  and  so  forth.  (A  parallel  is  the  visible  tendency 
within  the  field  of  education  and  training  to  teach  those  things  that  are  easiest  to  explain.) 
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Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Tasks  have  been  selected  for  training. 

Procedures 

1.  For  each  task  specify  for  the  job  environment: 

—  Conditions  of  performance. 

—  Behavioral  elements. 

—  Standards  of  performance. 

Analysis  of  Service  Models 

ITRO  Model.  Task  analysis  is  included  in  the  step  “Analyze  Job,”  which  specifies 
that  the  conditions,  initiating  cues,  standards,  and  elements  of  each  task  be  listed.  The 
manual  provides  a  Job  Data  Worksheet  that  can  be  used  for  this  purpose.  The  informa¬ 
tion  is  to  be  collected  in  the  same  manner  and  from  the  same  sources  as  data  for  the 
task  inventory.  This  can  be  done  at  the  time  of  the  original  inventory,  for  all  tasks,  or 
later  for  only  those  tasks  selected  for  training. 

Marine  Corps  Order  P1510.23B.  The  model  specifies  that  task  elements  be  identi¬ 
fied  along  with  tasks  when  job  analysis  is  conducted.  A  recommended  hierarchical  for¬ 
mat,  the  Job  Analysis  Sheet,  is  illustrated.  However,  standards,  conditions,  and  initiating 
cues  are  not  recorded  on  this  Job  Analysis  Sheet,  but  are  “preserved  to  be  used  during 
step  1.3,  Construct  Job  Performance  Requirements  and  Measures.”  During  step  1.3,  they 
are  written  down  on  a  different  form,  the  Consolidated  Job  Data  Sheet,  along  with  some 
other  information  concerning  the  task. 

Air  Force  Pamphlet  50-58.  The  model  specifies  that  tasks  be  divided  into  “sub¬ 
tasks”  as  part  of  job  analysis,  but  no  other  task  analysis  is  done  until  tasks  are  selected 
for  training.  Subtasks  are  identified  to  create  task  units  nearly  equal  in  complexity  and 
amount  of  implied  activity,  for  ease  of  analysis  later.  After  tasks  are  selected,  elements 
(“actions”),  cues,  standards  (“proficiency  requirements”),  time  required,  and  other  aspects 
of  the  task,  such  as  precautions  to  observe,  are  recorded  in  any  one  of  four  general  for¬ 
mats  shown  in  the  manual. 

The  AFP  50-58  model  does  not  discuss  task  conditions  per  se.  In  the  illustration  of 
the  use  of  a  recommended  format  for  task  description,  however,  two  of  the  items  are 
similar  to  conditions.  Under  the  heading  “Activity  Support  Elements”  for  a  task  involving 
the  use  of  a  calculator,  are  listed  the  calculator,  operator’s  manual,  list  of  values,  etc. 
Under  the  heading  “Support  Information”  are  listed  such  items  as  “Assume  values  are 
dollars  .  .  and  “Assume  prior  orientation  to  calculator.”1  In  this  latter  case,  however, 
it  appears  that  the  description  of  the  task  as  performed  on  the  job  is  beginning  to  be 
confounded  with  a  description  of  the  conditions  under  which  it  will  be  trained. 

Commentary.  The  procedures  suggested  in  the  three  models  do  not  differ  markedly. 
The  use  of  any  model  could  be  expected  to  result  in  an  adequate  definition  of  the  task. 


Construction  of  Job  Performance  Measures 

After  tasks  have  been  selected  for  training,  a  test  may  be  constructed  for  each  task 
to  serve  as  a  means  of  keeping  training  faithful  to  job  requirements,  as  a  means  of 


'AFP  50-58,  Vol.  II,  p.  2-51,  Fig.  2-38. 
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evaluating  training  and  the  design  process,  or  both.  To  serve  either  purpose,  the  test,  or 
Job  Performance  Measure,  should  possess  high  predictive  validity;  that  is,  performance  on 
the  test  should  be  highly  related  to  performance  of  the  task  itself.1  If  the  test  is  to 
serve  as  a  means  for  keeping  training  faithful  to  job  requirements,  the  behaviors,  condi¬ 
tions,  and  standards  of  the  Job  Performance  Measure  must  resemble  those  of  the  actual 
task  as  closely  as  is  practical.  If  the  test  is  to  be  used  to  evaluate  training,  it  can  con¬ 
ceivably  vary  in  fidelity  as  long  as  its  predictive  validity  is  maintained. 

To  evaluate  training  design,  Job  Performance  Measures  are  administered  to  persons 
in  training,  or  to  persons  after  they  have  reached  the  job.  Although  the  latter  method 
may  give  a  more  accurate  measure,  it  is  also  more  costly.  If  trainees,  or  graduates,  pass 
the  Job  Performance  Measure,  the  training  is  considered  successful.  If  not,  the  training 
cannot  be  considered  adequate,  regardless  of  trainees’  previous  performance  on  within- 
course  achievement  tests. 

In  attempting  to  keep  training  requirements  faithful  to  job  requirements,  the  Job 
Performance  Measures  serve  as  a  connecting  link  between  the  tasks  and  the  objectives  of 
training.  In  later  ISD  steps,  the  tasks  identified  by  job  analysis,  and  selected  for  training, 
must  be  translated  into  training  objectives  and  tests  of  those  objectives.  Some  degrada¬ 
tion  in  fidelity  can  be  expected  at  that  point:  job  conditions  cannot  always  be  repro¬ 
duced  in  training;  standards  may  need  to  be  lowered  for  training;  perhaps  only  part  of 
a  task  will  be  judged  to  require  training.  In  this  translation  from  job  requirements  to 
training  requirements,  there  will  be  much  opportunity  for  error.  In  the  name  of  adjust¬ 
ing  to  training  conditions  or  constraints,  critical  aspects  of  task  performance  for  which 
training  was  seen  as  necessary  can  virtually  disappear.  Job  Performance  Measures  can 
serve  as  standards  for  preventing  such  extreme  degradation. 

After  the  measures  have  been  constructed  and  validated,  objectives  can  be  derived 
directly  from  them,  and  achievement  tests  for  the  course  can  be  derived  in  turn  directly 
from  the  objectives.  Since  the  Job  Performance  Measures  have  already  been  determined  to 
be  good  predictors  of  task  performance,  then  objectives  and  tests  which  resemble  the 
Job  Performance  Measures  closely  can  be  expected  to  lead  to  successful  task  performance. 
Although  the  capacity  of  the  Job  Performance  Measures  to  help  achieve  this  end  is 
subject  to  how  well  they  are  adhered  to,  generally  the  existence  of  such  a  model  can  be 
expected  to  inhibit  departures  from  job  requirements. 

To  serve  their  purposes,  Job  Performance  Measures  must  be  constructed  before  objec¬ 
tives  or  tests  are  developed.  To  provide  an  independent  assessment  of,  or  point  of 
reference  for,  training,  they  must  be  independent  of  training.  If  they  were  not  constructed 
until  after  training  objectives  had  been  formulated,  it  would  be  nearly  inevitable  that  they 
would  reflect  training  to  some  degree.  To  whatever  extent  training  behavior,  standards, 
and  conditions  are  incorporated  in  the  Job  Performance  Measures,  they  are  that  much 
less  capable  of  measuring  the  job  adeouacy  of  the  training  product. 


1  In  traditional  test  construction,  Criterion-based  validity  has  been  viewed  as  the  most  desirable  kind 
of  validity  to  demonstrate.  Here  criterion-related  validity  would  be  a  statistical  demonstration  of  the 
relationship  between  the  test  and  the  ultimate  criterion,  actual  performance  of  the  task  on  the  job. 
Where,  however,  the  test  under  consideration  is  a  performance  test  that  is  content  valid,  there  is  usually 
no  better  criterion  of  task  performance  available.  In  the  absence  of  another  criterion  of  known  high 
validity,  it  is  meaningless  to  pursue  the  issue  of  criterion-related  validity.  In  such  situations,  another 
variety  of  predictive  validity— so-called  corcurrent  validity— can  be  sought,  that  is,  the  capacity  of  the 
test  to  distinguish  performers  who  are  acknowledged  to  be  effective  or  highly  experienced  from  those 
who  are  seen  as  ineffective  or  who  are  novices. 
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Not  all  models  of  ISD  include  the  construction  of  Job  Performance  Measures.  The 
alternative  to  deriving  objectives  and  tests  from  Job  Performance  Measures  is  to  derive 
them  directly  from  descriptions  of  tasks,  and  this  is  what  some  models  prescribe.  It  is 
difficult  to  foresee  exactly  how  training  development  is  affected  by  doing  without  a 
model  or  means  of  evaluation.  Certainly  the  relation  between  training  and  job  perform¬ 
ance  may,  in  some  cases  at  least,  be  so  obvious  that  the  construction  of  a  Job  Perform¬ 
ance  Measure  would  seem  superfluous.  Considerable  effort  is  involved  in  the  construction 
of  these  measures,  which  may  not  be  justified.  Nevertheless,  a  ready  means  for  deter¬ 
mining  whether  training  is  meeting  its  goals  would  appear  to  be  virtually  indispensable 
to  a  systematic  approach  to  training  design  and  development. 

Summary  of  Requirements 

Prerequisite  Conditions 

1.  Tasks  selected  for  training  are  listed. 

2.  Training  requirements  for  these  tasks  have  not  been  identified. 

Procedures 

1.  Construct  a  test  for  measuring  the  performance  of  each  task  selected 
for  training. 

2.  Validate  each  Job  Performance  Measure  to  insure  that  it  predicts  task 
performance. 

Analysis  of  Service  Models 

ITRO  Model.  A  Job  Performance  Measure  is  to  be  constructed  for  each  task  selected 
for  training.  The  model  discusses  predictive  validity,  and  states  that  it  cannot  be  determined 
if  there  is  no  way  to  test  the  task  under  actual  job  conditions.  In  such  cases,  the  model 
states,  the  physical  fidelity  of  the  Job  Performance  Measure  will  be  used  as  the  criterion 
of  its  adequacy.  The  process  of  measuring  the  physical  fidelity  of  the  Job  Performance 
Measure  is  termed  “verification.”  However,  although  the  model  distinguishes  between 
validation  and  verification  at  some  points,  it  confounds  them  at  others,  and  it  does  not 
adequately  explain  how  validation  or  verification  is  to  be  accomplished.  For  example,  it 
does  not  make  explicit  that,  in  the  validation  process,  task  performance  is  used  as  the 
validation  criterion,  while,  in  the  verification  process,  t|e  physical  fidelity  of  the  test 
itself  is  judged,  not  task  performance.  '  * 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  model  states  that  Job  Perform¬ 
ance  Measures  must  be  constructed  and  validated.  It  specifies  that  the  validity  of  the 
tests  should  be  higher  for  tasks  in  which  inadequate  performance  has  more  serious  con¬ 
sequences. 

Air  Force  Pamphlet  50-58.  This  step  is  not  included  in  lISD,  nor  is  any  equivalent 
step.  In  a  later  step,  objectives  will  be  derived  directly  from  tasks  and  task  elements. 

Commentary.  Despite  some  lack  of  clarity,  the  ITRO  model’s  explanation  of  how 
to  construct  and  validate  Job  Performance  Measures  would  appear  sufficient  to  enable  a 
test  designer  to  produce  them.  The  Marine  Corps  model,  which  severely  abbreviates  the 
ITRO  explanation,  and  combines  Job  Performance  Measure  construction  with  certain 
task  analysis  activities  (“Construct  Performance  Requirements”)  in  a  single  step,  would 
not  appear  altogether  adequate.  Perhaps  significantly,  the  Marine  Corps  model  does 
not  specify  later  that  objectives  be  derived  from  the  Job  Performance  Measures,  but 
rather  from  the  Job  Performance  Requirements,  which  are  essentially  records  of  task 
analysis.  The  role  of  the  Job  Performance  Measure  in  the  Marine  Corps  model  is  thus 
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somewhat  unclear.  As  noted,  the  AFP  50-58  model  objectives  are  also  derived  directly 
from  task  analysis  information.  In  this  respect,  the  AFP  50-58  model  is  consistent  when 
it  omits  the  construction  of  Job  Performance  Measures.  At  the  same  time,  this  omits 
from  the  model  a  significant  means  of  measuring  the  validity  of  instructional  programs. 


Selection  of  Setting 

A  tenet  of  ISD  is  that  training  should  be  provided  in  the  most  cost-effective  setting. 
Every  task  previously  selected  for  training  must  now  be  assigned  to  a  particular  location 
and  situation  in  which  training  will  occur.  Training,  may  be  conducted  in  one  or  in  a 
combination  of  situations:  at  resident  schools  and  training  centers,  at  installation  support 
schools,  and  through  a  variety  of  formal  on-the-job  procedures.  The  latter  category 
includes  the  use  of  such  techniques  as  self-teaching  lessons  and  job  aids  (proceduralized 
manuals  and  performance  guides). 

An  obvious  prerequisite  to  carrying  out  this  step  is  the  freedom  to  assign  tasks  to 
any  of  several  settings.  Without  a  range  of  options,  there  can  be  no  true  selection.  When 
only  one  setting  is  available,  there  is  the  danger  of  having  to  assign  an  excessive  number 
of  tasks  inappropriate  to  the  setting  or  to  forego  training  for  those  tasks  altogether.  It 
follows  that  a  variety  of  settings  must  be  available  and  that,  within  each  military  service, 
the  selection  of  setting(s)  must  be  made  at  an  organizational  level  which  has  the  authority 
to  implement  its  decision. 

Another  prerequisite  is  the  availability  of  information  about  the  costs  of  training  in 
the  alternative  settings.  Costs  to  be  examined  include  not  only  such  standard  considera¬ 
tions  as  equipment,  personnel,  facilities,  and  supplies,  but  also  the  indirect  consequences 
related  to  particular  training  strategies-for  example,  the  effects  of  different  experience 
mixes  (trained  and  untrained  men)  in  the  force  structure  when  training  is  conducted  in 
operational  units,  the  cost  in  productivity  to  experienced  incumbents  serving  as  instructors 
in  on-the-job  training,  and  the  cost  of  different  settings  as  a  function  of  trainee  charac¬ 
teristics  (e.g.,  level  of  maturity  and  capacity  for  self-study).  Thus,  one  study  found  an 
additional  year  of  education  to  be  associated  with  about  a  10%  reduction  in  estimated 
OJT  costs,  and  an  additional  10  points  of  measured  mental  ability,  with  about  a  6% 
reduction.1 

It  is  clear  that  this  step  involves  an  interface  with  other  components  of  the  opera¬ 
tional  system,  beyond  the  training  subsystem.  The  impact  of  the  size  of  the  partially 
trained  component  in  the  force  structure  and  the  impact  on  the  operational  unit  of  the 
amount  of  time  and  resources  devoted  to  training  cannot  simply  be  ignored.  A  systems 
approach  implies  that  no  part  of  the  system  is  free  of  the  influence  of  the  others.  To 
assign  tasks  to  different  settings  in  such  a  way  as  to  maximize  total  system  effectiveness, 
then,  requires  knowledge  of  how  alternative  assignment  patterns  will  affect  the  total 
system. 

Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Tasks  can  be  assigned  to  any  of  several  settings. 

2.  Information  is  available  on  costs  of  training  in  different  settings. 

3.  Information  is  available  on  effects  of  training  in  different  settings  on 
total  system  effectiveness. 


1  Gay,  Robert  M,,  Estimating  the  Cost  of  On-the-Job  Training  in  Military  Occupations:  A  Meth¬ 
odology  and  Pilot  Study,  Rand,  R-1351-ARPA,  April  1974. 
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Procedures 


1.  Assign  each  task  or  group  of  tasks  to  its  appropriate  training  setting. 
Analysis  of  Service  Models 

ITRO  Model.  The  ITRO  model  discusses  five  settings  to  which  tasks  may  be  assigned 
job  performance  aid,  self-teaching  exportable  package,  formal  on-the-job  training,  support 
school  in  operational  setting,  and  resident  school.  The  development  of  job  performance 
aids  in  lieu  of  training  is  recommended  whenever  task  delay  tolerance  and  environmental 
conditions  allow  use  of  the  aids,  when  the  task  does  not  require  an  aid  of  outsize  pro¬ 
portions,  and  when  successful  performance  is  not  dependent  on  a  high  degree  of  physical 
skill.  Otherwise,  each  of  the  other  four  settings  is  considered,  in  the  order  listed.  It 
is  recommended  that  choices  be  made  by  first  considering  non-cost  factors,  then  con¬ 
sulting  local  managers  to  revise  selections  as  necessary  in  light  of  costs. 

The  ITRO  model  acknowledges  that  ordinarily  not  enough  information  is  available 
on  costs  and  effectiveness  to  allow  rules  to  be  formulated  to  prescribe  specific  settings. 
Instead,  guidelines  are  given  that  “should  prove  helpful  in  making  logical  decisions.”1 
Some  of  the  factors  discussed  are  amount  of  supervision  required  for  a  task,  number  and 
locations  of  trainees,  learning  decay  rate,  and  training  resources  availability.  The  model 
points  out  that  the  expense  of  maintaining  a  program  of  instruction  is  not  the  only  cost 
factor;  development  and  implementation  must  be  considered  as  well  (e.g.,  since  the 
Services  differ  in  the  extent  to  which  on-the-job  training  systems  are  already  in  effect, 
they  might  rightly  differ  on  selection  of  setting  for  the  same  task). 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  model  indicates  the  same  five 
settings  as  the  ITRO  model,  and  specifies  that  they  be  considered  in  the  same  order. 
Resident  instruction  is  regarded  as  the  most  expensive  setting,  to  be  used  only  when  no 
other  will  suffice.  Some  advantages  and  disadvantages  are  also  discussed  for  job  perform¬ 
ance  aids  and  on-the-job  training. 

Air  Force  Pamphlet  50-58.  The  model  acknowledges  that  the  choice  of  setting 
will  seldom  be  left  to  the  training  developer,  and  provides  no  procedures.  In  interpreting 
this  treatment,  it  must  be  kept  in  mind  that  AFP  50-58  is  intended  primarily  for  the 
use  of  flying  training  and  technical  training  personnel  responsible  for  developing  resident 
instruction.  Air  Force  Manual  50-2,  which  describes  ISD  in  broader  terms,  states  that 
“instructional  system  designers  must  evaluate  the  various  alternatives  for  acquiring  quali¬ 
fied  personnel.  Among  the  alternatives  to  be  considered  are  the  relative  suitability  of 
selective  personnel  assignment,  OJT,  the  use  of  existing  resident  or  field  training  courses, 
and  the  development  of  new  resident  or  field  training  courses.  Generally,  a  requirement 
for  large  numbers  of  personnel  to  be  qualified  over  a  long  period  of  time  can  best  be 
satisfied  by  the  use  of  resident  courses.”2 

Commentary.  Neither  of  the  models  (ITRO  and  Marine  Corps)  that  suggest  pro¬ 
cedures  for  selection  of  setting  includes  a  methodology  for  determining  which  setting  is 
optimal.  If  costs  and  effectiveness  are  to  be  measured  and  compared,  the  training  designer 
will  need  to  provide  the  procedures.  The  essence  of  the  present  procedures,  then,  is  that 
formal  training  should  be  avoided  if  a  job  performance  aid  will  suffice,  and  that  settings 
other  than  resident  training  should  be  considered. 


'IPISD,  Block  1.5,  para.  2-4. 
JAFM  50-2,  para.  3-4. 
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Development  of  Training  Objectives  and 
Objectives  Hierarchies 


The  preceding  steps,  generally  referred  to  as  “front-end”  analysis,  have  focused  pri¬ 
marily  on  job  requirements.  The  development  of  training  objectives  represents  a  shift 
in  focus  and  is  the  first  step  in  designing  training  to  meet  these  requirements.  It  is  a 
pivotal  step  in  the  ISD  process  because  objectives  provide  the  bridge  between  performing 
a  task  and  learning  to  perform  a  task.  Objectives  are  descriptions  of  what  a  trainee  will 
be  able  to  do  following  instruction.  They  thereby  establish  what  behavior,  when  exhibited 
by  trainees,  will  be  accepted  as  evidence  that  instruction  was  successful.  In  turn,  they 
become  the  goals  for  training  and  the  determinants  of  test  and  instructional  content. 
Training  design  and  development  from  this  point  forward  will  be  carried  out  with  respect 
to  the  objectives  rather  than  to  the  job  tasks  themselves. 

Each  objective  describes  some  trainee  behavior  to  be  observed,  the  conditions  under 
which  it  will  occur,  and  the  standard  of  proficiency  that  will  be  considered  satisfactory. 

All  objectives  are  derived  from  the  tasks  selected  for  training.  They  describe  either  task 
performance  itself  or  behavior  which  demonstrates  knowledge  or  basic  skill  required  for 
task  performance. 

Objectives  which  describe  task  performance  are  given  the  name  terminal  (or  task, 
primary,  or  criterion)  objective.  In  managing  instruction,  tasks  are  sometimes  divided 
into  subtasks.  Objectives  which  correspond  to  the  performance  of  these  subtasks  are 
named  intermediate  (or  sub-,  enabling,  supporting,  or  secondary)  objectives.  Objectives 
that  refer  to  knowledge  or  basic  skills  that  mediate  the  performance  of  tasks  or  subtasks 
are  also  termed  intermediate  objectives. 

The  process  of  developing  objectives  from  tasks  selected  for  training  involves  several 
steps: 

•  Deciding  how  closely  capabilities  for  task  performance,  defined  in  terms 
of  behaviors,  conditions,  and  standards,  at  the  conclusion  of  training  shall 
match  the  requirements  of  the  job.  Those  specified  in  a  training  objective 
may  be  identical  to  those  identified  in  the  original  analysis  of  job  require¬ 
ments,  or  they  may  differ  either  because  of  constraints  in  the  training  situa¬ 
tion  or  because  it  is  not  efficient  to  attempt  to  bring  trainees  completely  up 
to  job  standards  through  formal  training. 

•  Identifying  objectives  in  a  process  referred  to  as  hierarchical  analysis.  Here 
a  task  is  successively  broken  down  into  its  component  parts  and  the  skills 
and  knowledge  that  are  necessary  to  learn  or  perform  each  part.  The  analysis 
is  taken  down  to  the  points  where  it  is  estimated  that  entering  trainees  would 
have  the  performance  capabilities.  The  product  of  this  analysis  is  a  specifi¬ 
cation  or  hierarchical  display  of  dependent  and  coordinate  relationships  within 
a  task  (terminal  objective),  among  subtasks  (intermediate  objectives),  and 
mediating  skills  and  knowledge  (intermediate  objectives)  for  which  training 
must  be  provided. 

•  Based  on  estimates  of  the  abilities  of  entering  trainees,  deciding  what  per¬ 
formance  capabilities  must  actually  be  developed  during  training.  Selecting 
a  task  for  training  has  implied  that  the  task  is  important  enough  that 
trainees  must  possess  some  degree  of  proficiency  in  it  following  training. 
Because  trainees  already  may  possess  the  ability  to  perform  all  or  part  of  a 
task  as  it  has  been  defined  for  training,  a  determination  is  made  of  which 
performance  capabilities  will  actually  require  instruction. 
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•  Based  on  analysis  of  the  task  capabilities  to  be  developed  during  training  and 
upon  estimates  of  abilities  of  entering  trainees,  deciding  what  knowledge  and 
basic  skills  must  be  provided  to  mediate  learning  and  performance.  Learning 
or  performing  a  task  may  require  the  prior  acquisition  of  certain  more  funda¬ 
mental  skills  (e.g.,  literacy  requirements,  mathematical  skills,  basic  electronics 
knowledge)  not  observable  or  identifiable  during  task  analysis.  Like  task 
performance  capabilities,  these  skills  may  exist  in  some  degree  in  the 
entering  trainee  population. 

Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Tasks  selected  for  training  are  listed. 

2.  Information  is  available  about  training  constraints  that  make  it  necessary 
to  modify  task  requirements. 

3.  Information  is  available  about  how  modification  of  task  requirements 
will  affect  training  efficiency. 

4.  Estimates  of  capabilities  of  entering  trainees  for  learning  and  performing 
each  task  are  available. 

Procedures 

1.  Specify  task  requirements  (behaviors,  conditions,  standards)  for  training. 

2.  Perform  hierarchical  analysis  of  tasks  to  identify  intermediate  training 
objectives. 

Analysis  of  Service  Models 

ITRO  Model.  The  ITRO  model  specifies  that  every  task  selected  for  training  be  used 
as  the  basis  for  an  objective.  This  is  accomplished  by  directly  translating  each  Job  Per¬ 
formance  Measure  into  a  terminal  objective.  This  procedure  automatically  establishes  the 
degree  of  fidelity  of  the  training  objective  to  the  job  task,  since  any  necessary  reduction 
in  fidelity  was  incorporated  into  the  Job  Performance  Measure. 

Next,  intermediate  objectives  are  derived  by  analyzing  the  terminal  objective  in  terms 
of  what  knowledge  and  skills  would  be  required  to  attain  it.  In  some  cases,  the  resultant 
intermediate  objectives  will  be  the  same  as  the  elements  of  the  task  itself.  In  other  cases, 
knowledge  and  skills  not  apparent  from  the  task  analysis  may  emerge  as  well,  in  a  learning 
hierarchy.  In  all  cases,  the  analysis  is  continued  only  until  intermediate  objectives  are 
derived  which  are  estimated  to  be  within  the  capabilities  of  the  entering  trainees.  This 
implicit  estimate  of  entry  behavior  will  later  be  verified  using  tests  derived  from  the 
objectives. 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  model  specifies  similar  procedures 
for  deriving  and  verifying  intermediate  objectives.  It  also  recognizes  that  knowledge  and 
skills  required  to  attain  the  objectives  but  not  evident  from  task  analyses  alone  may  emerge 
when  objectives  are  analyzed.  Regarding  the  development  of  terminal  objectives  and  their 
fidelity  to  job  tasks,  however,  the  Marine  Corps  model  offers  significantly  less  guidance 
than  the  ITRO.  The  introduction  to  the  design  phase  of  the  Marine  Corps  manual  states 
that  “each  task  or  task  element  selected  for  training”  will  become  a  terminal  objective. 

It  is  not  clear  whether  this  means  that  some  tasks  and  some  elements,  or  all  tasks  and 
some  elements,  will  be  used.  The  section  on  developing  objectives  (paragraph  310) 
specified  only  that  “the  terminal  objective  must  be  related  to  a  specific  Job  Performance 
'"Requirement,”  and  not  which  Job  Performance  Requirements  will  become  objectives. 
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No  further  guidance  on  this  question  is  offered.  Although  the  reader  is  directed  to  an 
appendix  for  further  guidance,  the  appendix  deals  primarily  with  the  clarity,  specificity, 
and  completeness  of  the  objectives  themselves,  and  does  not  include  a  procedure  for 
deriving  them  from  tasks. 

Nor  does  the  Marine  Corps  model  contain  a  procedure  for  establishing  standards 
and  conditions  for  training  objectives.  In  the  absence  of  any  guidance  to  the  contrary, 
it  might  be  inferred  that  the  standards  and  conditions  of  the  Job  Performance  Require¬ 
ment  with  which  the  objective  is  “directly  related”  should  be  used.  This  interpretation 
is  not  wholly  satisfactory,  however,  since  some  change  in  the  standards  or  conditions  is 
almost  inevitable— especially  for  combat  tasks.  The  lack  of  explicit  directives  for  deriving 
objectives  from  tasks  creates  the  possibility  that  products  from  different  training  develop¬ 
ers  will  vary  widely  in  quality. 

Air  Force  Pamphlet  50-58.  According  to  the  AFP  50-58  model,  objectives  are 
derived  directly  from  both  tasks  and  elements  of  tasks.  A  determination  is  made  for 
each  task  and  element  whether  practice  will  be  required  in  training  to  attain  the  standard 
set  for  it.  Each  task  or  element  for  which  practice  is  judged  necessary  is  considered  a 
“new”  skill  or  knowledge,  and  is  accordingly  translated  into  an  objective  (tasks  into 
“task  objectives,”  elements  into  “sub-objectives”).  Those  for  which  no  practice  is  judged 
necessary  are  considered  to  be  in  the  incoming  trainee’s  repertoire.  No  objectives  are 
developed  for  these. 

In  the  AFP  50-58  model,  then,  the  initial  estimate  of  entry  behavior  is  used  not 
only  to  establish  the  level  of  detail  of  intermediate  objectives,  but  also  to  delete  certain 
tasks  selected  for  training.  The  knowledge  and  skills  that  become  training  objectives  in 
the  AFP  50-58  model  thus  constitute  a  subclass  of  those  in  the  ITRO  and  Marine  models: 
ones  which  have  been  identified  as  (a)  requiring  a  degree  of  proficiency  unobtainable 
without  practice  in  training,  and  (b)  not  in  the  incoming  trainee’s  repertoire. 

Since  any  task  judged  to  be  within  the  entry  population’s  capabilities  is  eliminated 
from  further  consideration  for  the  proposed  course,  this  procedure  amounts  to  a  further 
selection  of  tasks  for  training— at  least  with  respect  to  the  deleted  tasks.  Thus  the  selec¬ 
tion  of  tasks  made  on  the  basis  of  task  priority  information  is  ultimately  subject  to 
deletions  made  on  the  basis  of  estimates  of  entry  behavior. 

A  “survey  test”  will  later  be  given  to  verify  this  estimate,  but  its  use  is  not  equiva¬ 
lent  to  the  later  ITRO/Marine  step  “Verify  Entry  Behavior.”  In  the  ITRO  model  (and 
possibly  also  in  the  Marine  Corps  model),  all  tasks  selected  for  training  are  subject  to  this 
method  of  verification.  In  the  AFP  50-58  model,  only  tasks  estimated  to  be  a  part  of 
entry  behavior  are  subject  to  verification.  Since  no  objectives  are  derived  from  the 
deleted  tasks,  no  tests  are  developed  to  measure  them.  The  process  will  thereby  identify 
only  underestimates.  Overestimation— the  judgment  that  trainees  are  able  to  perform  tasks 
which  they  really  cannot — will  not  be  revealed. 

It  is  not  clear  whether  the  AFP  50-58  procedures  identify  knowledge  and  skills 
that  are  required  to  attain  terminal  objectives  but  are  not  themselves  elements  of  job 
tasks.  At  one  point,  in  describing  the  possible  uses  of  the  optional  Instructional  Planning 
Worksheet,  the  manual  refers  to  “common-element”  objectives  as  though  they  belong  in 
that  category.  “Common-element  objectives  are  not  derived  from  specific  Job  Perform¬ 
ance  Requirements  or  Training  Requirements  recorded  on  the  Training  Data  Worksheet.”' 
This  would  suggest  the  existence  of  some  procedure  for  developing  objectives  beyond 
that  of  considering  each  task  and  element  listed  on  the  Training  Data  Worksheet. 


'AFP  50-58,  Vol.  Ill,  para.  l-4a(2). 
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A  little  further  in  the  manual,  however,  this  is  contradicted.  A  step-by-step  pro¬ 
cedure  is  given  for  deriving  such  objectives,  beginning  “(1)  To  identify  common-element 
objectives,  you  must  review  all  Training  Data  Worksheets  to  determine  the  skills  and 
knowledge  that  are  Training  Requirements.”1  There  is  a  column  on  the  Training  Data 
Worksheet  for  indicating,  for  each  task  or  element,  whether  it  calls  for  any  knowledge  or 
skill.  This  rightly  acknowledges  that  some  elements— perhaps  even  some  tasks— might 
require  neither.  The  instructions  for  doing  this,  however,  do  not  suggest  that  skills 
and  knowledge  are  being  identified  which  are  not  part  of  the  explicit  description  of  the 
task.  It  appears  more  likely  that  the  element  or  task  is  itself  considered  the  skill  or 
knowledge. 

There  is  also  a  Notes  column  on  the  Training  Data  Worksheet,  in  which  the  trainer 
is  instructed  to  explain  his  knowledge  classification  (or  skill  requirements).  Yet,  there  is 
no  further  guidance  to  suggest  that  the  explanation  in  the  Notes  column  be  converted 
into  a  separate  objective.  According  to  the  instructions  in  AFP  50-58,  any  objective 
derived  from  this  item  will  be  a  conversion  of  the  action,  conditions,  and  standard  listed 
for  the  element  or  task  on  the  Training  Data  Worksheet. 

For  deciding  how  closely  the  training  objectives  shall  match  job  tasks,  the  AFP  50-58 
model  prescribes  a  two-part  procedure.  The  conditions  and  standards  originally  deter¬ 
mined  for  the  task  or  element  are  to  be  used  for  the  objective  (with  necessary  allowances 
made  for  differences  in  conditions  between  the  job  and  training  environments).  A 
second  criterion  is  applied,  however,  before  the  standard  becomes  final.  The  standard 
for  the  objective  should  “reflect  the  proficiency  level  shown  on  the  Training  Data  Worksheet.”2 

Proficiency  levels  are  defined  as  “  the  amount  (extent)  of  knowledge  or  skill  required 
to  perform  a  task  or  activity  on  the  job.”3  Everything  that  follows  in  AFP  50-58,  how¬ 
ever,  makes  it  clear  that— at  least  as  far  as  training  design  is  concerned— proficiency  levels 
are  training  standards,  not  job  standards.  They  are  determined  on  the  basis  of  the  “train¬ 
ing  factors”  (task  priority  information).  They  indicate  how  proficient  the  graduate  is 
expected  to  be  upon  arrival  on  the  job.  (In  some  cases,  the  Air  Force  specifies  two  levels: 
one  the  desired  degree  of  proficiency,  the  other  a  lesser  degree  to  be  achieved  in  training 
when  the  desired  degree  cannot  be  reached  because  of  training  resource  or  other  con¬ 
straints.  The  unit  receiving  the  graduate  is  then  responsible  for  further  training.)  Pro¬ 
ficiency  levels  range  from  “extremely  limited”  to  “highly  proficient”  in  skill,  and,  for 
tasks,  from  knowledge  of  “nomenclature”  to  “complete  theory.” 

AFP  50-58  makes  clear  that,  when  this  proficiency  level  is  considered,  the  standard 
for  the  job  will  not  necessarily  become  the  standard  for  training.  This  is  not  wrong  per  se, 
but  according  to  the  AFP  50-58  model  the  job  standard  was  used  in  determining  which 
tasks  would  become  training  objectives  in  the  first  place.  The  criterion  by  which  certain 
tasks  and  elements  became  objectives,  and  others  did  not,  was  whether  the  job  proficiency 
requirement  could  be  met  by  incoming  trainees.  For  example,  if  the  job  task  was  to 
replace  a  certain  component  within  90  minutes,  and  it  had  been  estimated  that  incoming 
trainees  would  be  unable  to  do  this  without  practice  in  training,  that  action  became  the 
performance  part  of  an  objective.  If  this  task  was  performed  infrequently  and  was  not 
critical,  it  was  later  probably  assigned  knowledge  proficiency  level  “a”  and  skill  proficiency 
level  “1”.  These  would  indicate  that  only  nomenclature  and  simple  procedures  would  be 
taught,  and  that  the  trainee  would  graduate  with  “extremely  limited”  skill,  being  unable 
to  perform  the  task  to  “minimum  acceptable  levels  of  speed  or  accuracy.” 


1  Ibid.,  para.  3-2d(l). 

2 Ibid.,  para.  2-4f. 

3Op.  cit.,  Vol.  II,  para.  3-13a. 
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If  the  trainee  is  not  going  to  be  trained  to  meet  job  proficiency  requirements,  then 
the  question  arises  as  to  why  his  inability  to  meet  them  without  practice  in  training  is 
used  as  the  basis  for  choosing  objectives.  In  this  example,  the  trainee  is  unable  to  achieve 
the  task  standard  before  training,  and  he  is  unable  to  reach-  it  after  training.  Yet  his 
inability  to  achieve  it  before  training  was  the  reason  for  translating  this  task  into  a 
training  objective. 

The  hypothetical  example  is  not.  an  extreme  one.  AFP  50-58  specifies  only  the  fol¬ 
lowing  conditions  under  which  a  trainee  would  be  expected  to  reach  “minimum  acceptable 
levels  of  speed  or  accuracy”  in  training:  when  the  task  is  critical  or  semi-critical  and 
(a)  there  is  a  long  time  between  training  and  performance,  or  (b)  the  task  is  hard  to  learn 
on  the  job,  or  (c)  the  task  is  both  hard  to  learn  on  the  job  and  rarely  performed,  in  which 
case  it  must  be  critical,  not  just  semi-critical.  Many  tasks,  if  not  most,  fall  short  of 
these  criteria.  Furthemjore,  if  a  “large  majority  qf  the  incoming  trainees  have  directly 
relevant  prior  experience  on  most  elements  of  the  task  or  activity,”  the  proficiency  levels 
for  training  are  to  be  reduced  one  level— thus  widening  the  gap  between  job  standards  and 
training  standards  still  further.1  It  is  easy  to  see  how  prior  experience  could  enable 
trainees  to  meet  specified  standards  in  less  time,  but  it  is  not  clear  why  this  would  be  a 
reason  to  lower  the  standards. 

Commentary.  Of  the  three  models  under  discussion,  the  ITRO  provides  the  most 
comprehensive,  explicit,  straightforward  procedures  for  translating  job  requirements  into 
training  objeatives.  The  Marine  Corps  model  is  unclear  regarding  how  closely  the  objec¬ 
tives  must  match  job  requirements.  The  AFP  50-58  model  permits  the  omission  of  any 
task  or  tasks  which  the  trainees  are  estimated  to  know,  without  testing  the  estimate.  Either 
of  these  “loopholes”,  as  it  were,  might  encourage  what  the  Marine  Corps  model  itself 
rightly  identifies  as  a  “weak  practice”— writing  learning  objectives  to  match  what  is  being 
taught,  rather  than  to  meet  job  requirements. 


Development  of  Achievement  Tests 

When  training  objectives  have  been  specified,  it  becomes  possible  to  develop  objective- 
referenced  achievement  tests.  Such  tests  can  be  used  for  many  purposes,  such  as  deter¬ 
mining  the  capabilities  of  the  entry  population,  determining  the  effectiveness  of  training, 
and  diagnosing  student  performance  during  training.  They  may  take  a  variety  of  forms, 
depending  on  the  nature  of  the  objective  being  measured:  performance  tests  can  be  used 
to  assess  behavior,  paper-and-pencil  tests  to  assess  knowledge,  and  so  forth. 

A  particular  requirement  of  the  ISD  model  is  that  tests  be  developed  directly  from 
objectives,  rather  than  from  the  content  of  lessons.  There  are  several  reasons  for  this. 
Primarily  there  is  the  need  for  determining  whether  trainees  have  mastered  the  objectives. 
This  is  central  to  the  ISD  process.  Also,  the  procedure  helps  insure,  indirectly,  that  the 
content  of  a  lesson  will  support  its  objective,  since  it  must  prepare  the  trainee  to  pass  the 
test.  If  tests  are  based  instead  on  lesson  content,  there  is  nothing  to  insure  that  lessons 
will  pertain  to  objectives.  Similarly,  deriving  tests  from  objectives  is  expected  to  inhibit 
the  introduction  of  extraneous  material  into  lessons,  since  the  instructional  developer’s 
goals  are  clear. 

Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Training  objectives  have  been  specified. 

2.  Instructional  materials  have  not  been  developed. 

1  Ibid.,  Fig.  3-14. 
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Procedures 

1.  Determine  appropriate  types  of  tests  based  on  characteristics 
of  objectives. 

2.  Construct  tests  to  assess  attainment  of  objectives. 

Analysis  of  Service  Models 

1TRO  Model.  At  least  one  achievement  test  item  for  an  end-of-training  test  must  be 
derived,  from  the  Job  Performance  Measures,  for  each  terminal  and  each  major  intermedi¬ 
ate  objective.  It  is  not  clear  whether  more  than  one  item  per  objective  is  recommended. 
Guidance  is  not  provided  on  sampling  requirements  for  testing  the  transfer  and  applica¬ 
tion  of  information,  concepts  and  rules,  and  skill.  The  actions,  cues,  conditions,  and 
standards  of  these  items  are  to  be  identical  to  those  of  the  Job  Performance  Measures, 
unless  additional  constraints  in  the  training  situation  require  a  reduction  in  fidelity.  In 
addition,  entry  tests  (to  determine  qualifications  for  entering  the  course),  pretests  (to  be 
used  for  trainee  placement  within  the  course),  and  other  within-course  tests  are  developed 
if  needed. 

Marine  Corps  Order  P1510.23B.  Guidance  similar  to  the  ITRO  model  is  presented 
in  an  abbreviated  form.  A  test  item  is  to  be  prepared  for  every  terminal  objective,  and 
every  enabling  objective. 

Air  Force  Pamphlet  50-58.  The  model  requires  first  considering  what  resources  are 
available  for  test  development,  and,  if  tests  cannot  be  developed  for  all  objectives,  selec¬ 
ting  the  most  important,  according  to  a  weighting  formula  provided.  The  model  recom¬ 
mends  developing  a  Course  Criterion  Test  with  items  derived  from  task-level  objectives, 
a  Diagnostic  Test  (a  pretest  for  placement  purposes)  with  items  derived  from  intermediate 
objectives,  and  a  Survey  Test  comprised  of  both.  The  purpose  of  the  Survey  Test  is  to 
verify  the  assumptions  about  student  entry  behavior  made  when  objectives  were  developed. 

Commentary.  All  of  the  models  require  that  tests  be  developed  from  objectives, 
rather  than  from  the  content  of  lessons.  All  provide  some  information  about  test  con¬ 
struction,  ranging  from  descriptions  of  different  types  of  tests  to  discussions  of  such 
conventional  topics  as  reliability  and  validity.  The  ITRO  and  Marine  Corps  models,  how¬ 
ever,  provide  little  guidance  concerning  what  an  adequate  test  of  an  objective  would  be. 
They  do  not  indicate  how  item  content  and  form  are  to  be  selected  or  how  content  is 
to  be  sampled  to  test  for  the  transfer  and  application  of  skill  and  knowledge.  All  models 
lack  procedures  for  maintaining  congruence  between  the  actions  and  processes  implied 
in  an  objective  (e.g.,  remember  information,  use  information,  remember  concepts,  use 
concepts)  and  the  actual  requirements  imposed  by  test  items.1 


Identification  of  Entry  Behavior 

As  soon  as  tests  have  been  developed,  it  is  possible  to  verify  the  estimates  of  trainee 
entry  behavior  which  provided  the  basis  for  deriving  objectives  in  an  earlier  step.  At 
that  time,  tasks  were  analyzed  to  identify  their  component  parts  and  the  skills  and  knowl¬ 
edge  necessary  to  perform  each  part.  In  order  to  avoid  reducing  objectives  to  trivial 


1  For  a  procedure  to  assess  the  consistency  of  requirements  among  objectives,  instructional  activi¬ 
ties,  and  tests,  see  Wulfeck,  Wallace  H.,  II,  et  al,  The  Instructional  Quality  Inventory:  1,  Introduction 
and  Overview,  Special  Report  79-3,  Navy  Personnel  Research  and  Development  Center,  1978. 
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levels,  an  estimate  was  made  of  the  performance  capabilities,  skills,  and  knowledge  enter¬ 
ing  trainees  would  already  possess.  That  estimate  is  now  verified,  before  instructional 
development  proceeds. 

Since  individuals  who  will  be  trained  with  this  instruction  probably  are  not  available 
at  this  point,  trainee  entry  behavior  is  determined  by  measuring  performance  of  a  group 
that  is  representative  of  the  entry  population  in  aptitude,  experience,  previous  training,  etc. 
If  any  objectives  are  found  to  be  already  a  part  of  the  incoming  trainees’  repertoires,  they 
can  be  deleted  from  the  proposed  course  at  this  time.  On  the  other  hand,  the  representa¬ 
tive  trainees  may  fail  to  attain  objectives  that  had  been  estimated  to  be  within  their  capa¬ 
bilities.  These  objectives  are  then  analyzed  further  to  derive  lower  order  objectives; 
tests  are  developed  for  these  subordinate  objectives,  and  the  new  estimate  of  entry  behavior 
is  verified  by  administering  the  tests  to  another  representative  sample  of  trainees.  This 
cycle  is  continued  until  a  set  of  objectives  is  obtained  that  is  consistent  with  the  verified 
estimate  of  entry  behavior.  It  is  this  refined  set  of  objectives  that  will  be  used  in  subse¬ 
quent  ISD  steps. 

Later,  after  the  instructional  program  has  been  implemented,  entry  tests  and  pretests 
can  be  used  to  measure  entry  capabilities  directly.  If  these  capabilities  vary  from  the 
final  estimate  made  in  the  present  step,  objectives  can  be  added  or  deleted  as  necessary  at 
that  time. 

Entry  behavior  is  also  assessed  to  diagnose  trainee  needs  when  instruction  is  individ¬ 
ually  prescribed.  Here  objectives  are  not  modified,  but  rather  training  content  is  adjusted 
according  to  individual  requirements  for  attaining  the  objectives. 

Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Objectives  have  been  derived  through  hierarchical  analysis  of  tasks. 

2.  Tests  are  available  to  measure  objectives. 

Procedures 

1.  Identify  sample  that  is  representative  of  trainees. 

2.  Administer  tests  to  sample,  and  determine  accuracy  of  earlier  estimate 
of  entry  behavior. 

3.  Add  or  delete  objectives  as  indicated  by  test  results,  and  repeat  cycle. 

Analysis  of  Service  Models 

ITRQ  Model.  The  end-of-course  test  is  to  bn  administered  to  a  sample  of  25  to  30 
trainees  representative  of  those  who  will  undergo  the  proposed  instruction.  The  results 
are  then  used  to  revise  the  objectives.  Objectives  which  the  sample  have  already  mastered 
are  deleted.  Those  which  were  erroneously  assumed  to  have  been  mastered  are  analyzed 
to  a  lower  order;  tests  are  then  administered  to  another  representative  group,  and  the 
cycle  is  repeated  until  the  objectives  match  the  trainee  entry  level.  When  the  trainees 
differ  on  whether  they  have  already  mastered  an  objective,  it  is  recommended  that  pre¬ 
tests  be  developed  to  identify  the  entering  trainees  who  do  not  need  instruction.  If  it 
appears  that  the  prerequisites  for  entry  into  the  instruction  are  not  sufficient  to  guarantee 
that  trainees  will  possess  necessary  skills  or  knowledge,  an  entry  test  may  be  developed 
as  well. 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  procedures  are  similar  to  those 
of  the  ITRO  model,  although  test  sample  size  is  not  specified.  In  addition,  the  Marine 
model  specifies  that  all  objectives  corresponding  to  skills  that  all  Marines  in  an  MOS 
must  perform  correctly  (“critical  skills”)  shall  be  retained  in  the  instruction,  regardless  of 
the  performance  of  the  trainees  in  the  representative  sample. 
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Air  Force  Pamphlet  50-58.  The  AFP  50-58  procedures  are  also  basically  the  same 
as  those  in  the  ITRO  model.  The  recommended  minimum  sample  size  for  verifying 
entry  behavior  is  ten.  There  appears  to  be  a  contradiction  between  the  guidelines  for 
using  test  results  and  the  manner  in  which  the  objectives  were  developed.  It  will  be 
recalled  that  tests  were  developed  only  for  skills  and  knowledge  that  were  estimated 
not  to  be  in  the  entry  population’s  repertoire.  Yet  the  model  specifies  that  instruction 
should  be  increased  as  needed  whenever  the  students  in  the  representative  sample  have 
greater  difficulty  than  anticipated  in  exhibiting  the  task  behaviors.  If  the  behaviors 
were  estimated  to  be  outside  their  repertoires,  then  by  definition  it  was  anticipated  the 
students  would  not  exhibit  them  at  all. 

Commentary.  Except  for  the  contradiction  mentioned  in  AFP  50-58,  the  procedures 
of  any  one  of  the  models,  if  followed,  would  appear  to  be  effective  in  adjusting  the 
objectives  to  match  trainee  capabilities. 


Classification  of  Objectives  and 
Selection  of  Instructional  Activities 

Before  instruction  is  written— even  before  any  instructional  medium  is  selected— the 
specific  kinds  of  activities  necessary  to  provide  for  learning  are  identified.  In  later  steps, 
media  will  be  chosen  and  instructional  materials  developed  to  support  these  activities. 

Some  activities  are  recognized  as  appropriate  to  all  types  of  instruction,  (e.g.,  inform 
the  learner  of  the  objective,  elicit  his  performance,  provide  feedback).  The  most  effective 
instructional  activities,  however,  also  differ  according  to  the  nature  of  the  capability  to 
be  acquired,  such  as  information,  cognitive  skill,  or  motor  skill.  Thus,  information  is 
acquired  more  readily  if  trainees  are  required  to  restate  it  in  a  context  of  related  informa¬ 
tion;  visual  discriminations  are  learned  more  readily  when  critical  stimuli  are  presented 
in  different  surrounds;  acquisition  of  motor  skill  requires  practice,  and  so  forth. 

Before  instructional  activities  are  selected,  objectives  are  classified  according  to  the 
type  of  capability  they  represent.  Information  about  appropriate  activities  to  promote 
learning  for  each  type  of  capability  is  also  essential.  When  objectives  or  groups  of 
objectives  have  been  classified,  and  the  corresponding  types  of  instructional  activities 
identified,  the  activities  themselves  are  specified  (e.g.,  “Display  varing  views  of  the 
ships.  .  .  .  Provide  three  separate  opportunities  to  disassemble  the  servo-mechanism”). 
These  specifications  will  later  be  used  to  determine  appropriate  media  and  the  content 
of  instruction. 

The  choice  of  activities  and  the  degree  of  detail  required  in  specifying  them  will  be 
greatly  influenced  by  the  nature  of  the  behavioral  requirements  inherent  in  the  objectives 
and  how  apparent  they  are.  Some  requirements  will  be  self-evident  in  the  conditions  and 
standards  of  the  objective.  In  other  instances  it  will  be  necessary  to  analyze  the  objec¬ 
tive  (and  perhaps  be  more  specific  in  stating  conditions  and  standards)  before  its  behav¬ 
ioral  requirements  can  be  completely  identified.  When  requirements  are  not  immediately 
obvious,  instructional  activities  must  be  specified  in  greater  detail  to  provide  sufficient 
guidance  for  selecting  media  and  developing  instructional  materials  in  later  steps. 

Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Information  is  available  about  types  of  instructional  activities  appropriate 
to  acquiring  different  types  of  capabilities. 


Procedures 


1.  Classify  each  objective  or  group  of  objectives  according  to  type 
of  capability. 

2.  Specify  instructional  activities  for  each  objective  according  to  type 
of  capability. 

Analysis  of  Service  Models 

All  three  models  recognize  four  instructional  guidelines  as  applicable  to  all  types  of 
learning:  inform  the  learner  of  the  objective;  provide  for  active  practice;  provide  guidance 
and  prompts;  provide  feedback  quickly.  The  models  differ  in  the  amount  of  additional 
guidance  given  for  specifying  instructional  events  for  different  kinds  of  learning,  and  also 
in  the  degree  to  which  such  specification  is  required. 

ITRO  Model.  Each  objective  is  to  be  classified  as  one  of  11  types  of  learning  (iden¬ 
tifying  symbols,  performing  gross  motor  skills,  learning  attitudes,  etc.).  Descriptions  and 
examples  of  each  learning  type  are  provided.  Next,  it  is  recommended  that  about  four 
learning  guidelines  be  selected  for  each  objective  from  among  those  specified,  the  number 
depending  on  type  of  skill  (e.g.,  “Make  the  learning  activities  relevant  by  making  them 
similar  to  real  life  tasks  that  the  student  will  be  performing  on  the  job.  ...  If  students 
make  incorrect  actions  or  begin  to  develop  bad  habits,  present  a  penalty  following  these 
improper  responses  until  they  disappear.  .  .  .  Display  distinctive  features  of  the  pattern”). 
If  these  prove  insufficient  to  assist  learning,  more  can  be  added  later.  Each  guideline  is 
then  converted  into  specific  instructional  activities  unique  to  the  objective  (e.g.,  “Call 
attention  to  the  differences  between  the  two  ships”). 

The  ITRO  model  requires  recording  the  type  of  learning  and  corresponding  activities 
on  the  Learning  Objective  Analysis  Worksheet  (although  the  format  illustrated  in  the 
manual  provides  space  only  for  the  learning  category  and  not  for  the  activities). 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  model  acknowledges  that  “learn¬ 
ing  activities  will  vary  depending  upon  the  type  of  objective  being  taught,”  but  does  not 
indicate  specific  types,  nor  any  learning  guidelines  specific  to  different  types.  It  does 
require  that  whatever  instructional  activities  are  specified  be  recorded  on  the  Consolidated 
Training  Data  Worksheet. 

Air  Force  Pamphlet  50-58.  The  model  states  that  “the  lesson  plan  developed  from 
the  POI  [Program  of  Instruction]  should  contain  a  complete,  concise  description  of 
instructional  events  for  each  stage  of  learning.”'  The  development  of  the  POI  is  part  of 
instructional  design,  while  the  lesson  plan,  apparently,  will  be  developed  by  another  party. 
Strictly  speaking,  then,  AFP  50-58  does  not  require  instructional  events  to  be  specified  as 
part  of  the  ISD  process.  However,  a  format  is  provided  for  such  specification— the  Instruc¬ 
tion  Planning  Worksheet— to  be  used  at  the  option  of  the  training  designer.  Furthermore, 
AFP  50-58  provides  guidelines  for  helping  the  student  in  six  different  types  of  learning. 
The  manual  discusses  the  relative  advantages  of  part-task  and  whole-task  practice,  dis¬ 
tribution  of  practice  and  rest,  feedback,  cues,  and  similar  instructional  factors. 

Commentary.  The  ITRO  model  presents  a  large  array  of  different  “learning  guide¬ 
lines,”  covering  11  types  of  learning,  from  which  instructional  designers  can  select,  while 
the  AFP  50-58  model  gives  more  prescriptive  guidance  and  explanation  for  a  smaller 
number  of  learning  types.  The  Marine  Corps  model  would  not  appear  to  provide  suffi¬ 
cient  guidance  or  direction  to  insure  that  this  step  is  carried  out. 


1  AFP  50-58,  Vol.  IV,  para.  4-3c. 
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Selection  of  Instructional  Methods 


After  instructional  activities  have  been  specified,  instructional  methods  or  delivery 
systems  are  selected.  These  are  the  ways  in  which  trainees  will  be  brought  into  contact 
with  the  instruction.  These  decisions  involve  such  considerations  as  instructional  group 
size  and  location  (e.g.,  large  group,  seminar,  individual  study);  degree  of  individualization 
of  curriculum  (e.g.,  fixed  curriculum,  remedial  loops,  branching  programs);  pacing  (group 
or  individually  paced);  management  of  the  course  (e.g.,  instructor-managed,  computer- 
managed,  self-managed,  media-controlled);  and  who  will  provide  one-to-one  tutoring  if 
required  (e.g.,  instructor,  peer,  assistant  instructor).  Obviously,  some  methods  are  inter¬ 
dependent;  it  would  be  impractical  to  combine  a  high  degree  of  individualization  with 
group  pacing,  for  example. 

Methods  are  selected  so  as  to  maximize  the  effectiveness  and  efficiency  of  instruction, 
given  the  particular  instructional  activities,  the  nature  of  the  trainee  population,  the  setting, 
and  administrative  requirements  and  constraints.  The  choice  of  methods  is  influenced  to 
a  large  degree  by  the  kinds  of  instructional  activities  specified  in  the  previous  ISD  step. 

For  example,  if  trainees  are  to  observe  a  live  fire  demonstration,  it  will  be  convenient 
to  bring  them  together  in  a  group;  if  they  are  to  practice  the  procedure  for  preparing  an 
accident  report,  individual  study  or  peer  instruction  might  be  more  appropriate.  The 
choice  of  methods,  in  turn,  will  influence  the  selection  of  media  in  the  next  ISD  step. 

If  self-paced,  individualized  instruction  is  specified,  for  example,  a  live  lecture  will  proba¬ 
bly  be  precluded,  but  a  videotaped  one  might  be  acceptable.1 

Within  the  constraints  of  the  factors  just  mentioned,  methods  are  chosen  on  the 
basis  of  efficiency.  Individualization,  self-pacing,  computer  management,  and  similar  methods 
have  the  potential  for  reducing  the  average  time  in  the  course,  thus  reducing  total  costs 
when  large  numbers  of  people  must  be  trained.  Use  of  such  methods  can  .aise  develop¬ 
ment  costs,  however,  and  may  not  be  efficient  for  training  smaller  numbers.  The  choice 
is  also  influenced  by  level  and  homogeneity  of  aptitude  and  experience  of  the  trainee 
population,  the  probable  life  of  the  course,  and  the  availability  of  facilities  and  equipment . 
Another  consideration  is  setting;  if  the  training  is  to  be  presented  on  the  job,  for  example, 
peer  instruction  is  often  appropriate,  while  computer-managed  instruction  might  be 
impractical. 

Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Setting  has  been  specified. 

2.  Trainee  characteristics  have  been  identified. 

3.  Instructional  activities  have  been  specified. 

4.  Information  is  available  on  how  the  costs  and  effectiveness  of  alternate 
methods  vary  for  specified  settings,  trainee  characteristics,  and  instruc¬ 
tional  activities. 

Procedures 

1.  Specify  the  methods  of  instruction  to  be  employed  for  each  objective  or 
group  of  objectives. 


1  In  the  different  ISD  models,  distinctions  between  instructional  activities,  media,  and  methods  vary 
considerably,  and  some  models  combine  certain  of  these  steps.  These  differences  are  largely  definitional 
and  are  to  be  expected,  given  the  interrelated  nature  of  these  elements  and  the  processes  through  which 
they  are  identified.  For  clarity,  these  elements  are  discussed  here  as  different  entities  originating  in  dis¬ 
cretely  different  steps. 
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Analysis  of  Service  Models 

ITRO  Model.  The  ITRO  model  includes  selection  of  methods  in  the  step  “Specify 
Instruction  Management  Plan  and  Delivery  System,”  the  same  step  in  which  media  are 
chosen  and  a  System  Master  Plan  is  developed.  The  model  suggests  that  self-paced, 
individualized  instruction  is  generally  more  effective  than  group-paced  instruction. 

Different  methods  of  managing  students  are  presented  for  both  group  and  individual 
modes.  A  glossary  of  methods  and  instructional  devices  is  provided. 

Marine  Corps  Order  P151Q.23B,  The  Marine  Corps  model  also  includes  methods 
selection  along  with  media  selection.  The  model  provides  a  matrix  indicating  some 
advantages  and  disadvantages  of  several  methods.  It  is  recommended  that  alternatives  to 
the  lecture  be  used  if  possible. 

Air  Force  Pamphlet  50-58.  The  AFP  50-58  model  contains  a  Methods  Directory 
describing  the  characteristics  of  several  methods  and  their  applicability  to  different 
settings.  The  model  notes  that  the  stage  of  learning  (early,  middle,  or  late)  will  influence 
the  choice  of  methods. 

Commentary.  All  of  the  models  describe  and  discuss  different  methods,  and  all 
encourage  the  consideration  of  alternatives  to  the  traditional  group-paced  classroom  lec¬ 
ture.  While  they  specify  the  requirements  which  must  be  met  (in  terms  of  setting,  group 
size,  etc.)  before  a  particular  method  can  be  used,  they  provide  little  information  on  which 
to  base  decisions  about  the  relative  effectiveness  of  different  methods,  either  in  general  or 
with  respect  to  their  use  with  different  trainee  populations.  Thus,  while  the  models  may 
be  sufficient  for  rejecting  inappropriate  methods,  they  do  not  appear  sufficient  for  selecting 
optimal  methods. 


Selection  of  Media 

After  instructional  activities  and  training  methods  have  been  specified,  and  before 
instructional  materials  are  developed,  it  is  necessary  to  identify  what  media  will  provide 
an  effective  and  efficient  means  for  presenting  the  subject  matter  to  the  trainee.  For 
example,  if  it  has  been  decided  that  trainees  should  understand  how  the  working  parts  of 
an  engine  operate,  it  must  now  be  determined  whether  an  operational  engine,  simulator, 
television  or  motion  picture,  still  picture,  diagram,  computer-generated  image,  or  other 
device  is  appropriate  and,  of  those  which  are  appropriate,  which  is  most  economical. 

The  nature  of  the  instructional  activities  already  specified  determines  which  media 
can  be  considered.  These  activities  were  selected  to  expose  the  trainee  to  stimulus  events 
and  provide  response  opportunities  that  would  promote  learning.  To  be  appropriate, 
media  must  allow  those  stimulus  events  and  response  opportunities  to  occur.  If  the 
instructional  activity  required  the  trainee  to  distinguish  between  naval  signal  flags,  color 
would  presumedly  be  required,  but  not  motion.  If  the  activity  was  to  distinguish  between 
target  returns  and  noise  on  a  radar  scope,  then  motion  would  be  required  but  not  color. 

Which  medium  is  the  most  economical  is  a  function  of  several  factors,  the  first 
being  the  cost  of  the  medium  itself.  Under  most  circumstances,  for  example,  color 
costs  more  than  black  and  white,  motion  costs  more  than  no  motion,  audiovisual  media 
cost  more  than  printed  forms.  The  previously  selected  methods  also  affect  the  choice 
of  medium.  For  example,  if  self-paeed  instruction  had  been  designated,  tape/ slide  rather 
than  lecture  would  be  used  to  present  verbal  information;  if  group  instruction  had  been 
designated,  lecture  could  be  used.  Among  other  factors  contributing  to  cost  are  existing 
investment  in  equipment,  number  of  personnel  to  be  trained,  and  availability  and  location 
if  production  facilities. 


The  procedure  is  to  determine  first  which  media  satisfy  the  requirements  imposed 
by  the  learning  activities  and  methods  that  have  been  specified,  and  only  then  to  consider 
costs.  One  prerequisite  to  selecting  media,  then,  is  information  about  the  suitability  of 
different  media  to  different  instructional  activities  and  methods.  Another  is  information 
about  media  costs  and  administrative  factors  that  affect  costs.  In  general,  media  are 
not  selected  on  the  basis  of  effectiveness  per  se  since  known  characteristics  that  would 
make  one  medium  more  effective  for  certain  instructional  activities  or  methods  are  likely 
to  have  been  incorporated  when  the  activities  were  specified. 

Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Training  designers  are  free  to  select  from  a  range  of  media. 

2.  Instructional  activities  have  been  specified. 

3.  Methods  of  training  have  been  specified. 

4.  Information  is  available  concerning  the  appropriateness  of  different  media 
to  implement  different  activities  to  be  used  in  conjunction  with  different 
methods. 

5.  Information  is  available  concerning  the  costs  of  different  media. 

Procedures 

1.  Determine  which  media  will  be  suitable  to  implement  the  instructional 
activities  and  methods. 

2.  Consider  relative  costs  of  media  determined  above  and  select  most  economical 
set  of  media. 

Analysis  of  Service  Models 

ITRO  Model.  A  two-part  procedure  is  specified  for  selecting  media.  In  the  first  part, 
matrices  are  utilized  to  select  a  set  of  possible  media,  based  on  the  stimuli  inherent  in  the 
learning  activities,  the  complexity  of  the  objective,  the  training  setting,  the  development 
site,  and  the  amount  of  available  funds.  In  the  second  part,  final  selections  are  made  on 
the  basis  of  cost,  practicality,  existing  investment  in  production  facilities,  resistance  to 
innovation,  and  characteristics  of  the  trainees  such  as  reading  ability.  The  first  part  of  the 
procedure,  then,  is  algorithmic,  while  the  second  part  is  judgmental. 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  model  states  that  the  use  of  media 
can  “substantially  enhance  the  learning  process”1  by  stimulating  more  of  the  students’ 
senses,  presumedly  in  comparison  to  a  lecture.  This  implies  a  slightly  different  definition 
of  “media”  from  that  of  the  ITRO  model.  In  the  ITRO  model,  the  “traditional  classroom” 
is  one  of  several  media.  The  Marine  Corps  model  treats  mediated  instruction  as  an  adjunct 
to,  or  substitute  for,  traditional  methods.  Perhaps  for  this  reason,  the  Marine  Corps  model 
does  not  treat  the  matter  of  media  selection  in  great  detail. 

A  selection  matrix  for  choosing  media  is  provided,  but  no  directions  for  its  use.2  The 
matrix  displays  six  types  of  learning  and  13  “Characteristics/Advantages  of  Media,”  such 
as  “simplicity,”  “large  classes,”  “realistic,”  and  “motivation.”  If  the  type  of  learning  is 
given,  the  matrix  will  yield  from  two  to  16  appropriate  media.  Final  choices  would  pre¬ 
sumably  then  be  made  in  the  light  of  the  characteristics  or  advantages  corresponding  to 
the  media  indicated.  To  use  this  matrix,  it  is  first  necessary  to  classify  each  learning  objec¬ 
tive  according  to  type  of  learning.  However,  there  is  no  explicit  requirement  in  the 

'MCO  P1510.23B,  para.  430.4d. 

2  Ibid.,  App.  G. 
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Marine  Corps  model  to  do  this,  nor  is  there  any  space  indicated  on  the  Consolidated 
Training  Data  Worksheet  to  keep  a  record  of  type  of  learning. 

Air  Force  Pamphlet  50-58.  A  media  selection  matrix  and  a  procedure  for  its  use 
are  presented.  The  first  step  is  to  classify  the  knowledge  or  skill  represented  by  each 
objective  as  either  concrete  (e.g.,  motor  skill,  classifying  actual  objects)  or  abstract 
(e.g.,  rules  using  language,  verbal  learning).  This  is  done  for  early,  middle,  and  late 
states  of  learning  for  each  objective. 

For  each  concrete  skill  or  knowledge,  the  second  step  is  to  decide  whether  media 
are  needed.  The  model  states  that  actual  job  conditions  and  equipment  are  preferable. 
Media  should  be  used  only  when  certain  specified  practical  and  instructional  constraints 
prevent  the  use  of  job  conditions.  The  next  step  is  to  identify  which  of  three  stimuli— 
auditory,  visual,  and  kinesthetic— are  involved  in  learning  the  skill  or  knowledge.  A 
guide  is  provided  to  aid  this  determination.  The  stimulus  (“presentation  mode”)  in  turn 
determines  the  media  options.  When  the  matrix  identifies  more  than  one  medium,  a 
choice  is  made  on  the  basis  of  group  size,  cost,  instructor’s  role,  and  other  administrative 
factors  listed  in  a  Directory  of  Media. 

For  each  abstract  skill  or  knowledge,  the  second  step  is  to  determine  which  of  four 
presentation  modes— auditory,  visual,  semi-motion,  or  interpersonal  simulation— is  required 
to  express  the  concept  associated  with  the  knowledge  or  skill.  If  the  concept  can  be 
expressed  by  spoken  or  written  word  alone,  no  medium  is  required.  Otherwise,  a 
matrix  is  used  to  determine  media  options,  based  on  the  specified  presentation  mode  and 
the  size  of  the  group  receiving  instruction.  If  the  matrix  indicates  more  than  one  medium, 
a  choice  is  made  on  the  basis  of  the  administrative  factors  listed  in  the  Directory  of  Media. 

Commentary.  The  appropriateness  of  different  media  to  a  given  objective  is  deter¬ 
mined  in  the  ITRO  model  by  considering  the  stimulus  and  response  requirements  of  the 
instructional  activities;  in  the  AFP  50-58  model,  by  considering  the  objectives  themselves; 
and  in  the  Marine  Corps  model,  primarily  by  the  type  of  learning.  In  all  cases,  the 
accuracy  of  the  determination  would  appear  to  depend  on  the  skill  of  the  analyst.  None 
of  the  models  provides  guidance  on  how  to  determine  stimulus  requirements— only  on 
how  to  match  them  to  media. 

The  detailed  procedures  in  the  ITRO  and  AFP  50-58  models  clearly  represent  a 
serious  attempt  to  encourage  users  to  consider  a  wide  range  of  media.  The  incomplete 
treatment  of  this  step  in  the  Marine  Corps  model  brings  into  question  whether  Marine 
Corps  trainers  are  actually  expected  to  do  likewise. 


Grouping  and  Sequencing  of  Instruction 

To  create  a  course  of  instruction  from  the  diverse  objectives  that  now  exist,  the 
next  step  is  to  determine  and  specify  how  they  will  be  grouped  and  sequenced.  An 
obvious  factor  that  affects  grouping  is  commonality  of  subject  matter.  Another  is  trans¬ 
fer  of  learning;  to  the  extent  that  seems  reasonable,  objectives  should  be  grouped  together 
when  some  transfer  of  learning  between  them  can  be  anticipated. 

Two  kinds  of  factors  affect  sequencing.  One  is  the  degree  of  dependency  between 
objectives.  If  learning  one  objective  depends  on  learning  another,  they  must  be  sequenced 
accordingly.  Where  no  dependency  exists,  more  latitude  is  available  and  the  process 
invokes  the  second  factor,  the  set  of  overall  sequencing  principles  chosen;  simple  to 
complex,  job  order,  familiar  to  unfamiliar,  most  difficult  first,  most  difficult  last,  etc. 
Unfortunately,  there  are  no  universal  principles.  In  fact,  a  rationale  can  often  be  found 
for  conflicting  strategies.  For  example,  sequencing  topics  on  the  basis  of  the  order  of 
job  performance  may  often  be  most  meaningful  from  the  point  of  view  of  the  learners, 


35 


yet  it  may  sometimes  result  in  giving  the  most  exposure  and  rehearsal  to  the  easiest  and 
least  important  aspects  of  job  performance.  At  present,  decisions  about  sequencing  of 
instruction  must  be  based  in  part  on  individual  judgment. 

In  addition  to  factors  that  pertain  to  learning,  grouping  and  sequencing  decisions 
are  also  subject  to  practical  and  administrative  factors  such  as  safety  and  availability  of 
equipment  and  facilities.  A  principal  characteristic  of  the  ISD  approach  is  that  decisions 
about  sequencing  and  grouping  are  made  as  much  as  possible  on  the  basis  of  learning 
factors,  rather  than  solely  on  the  basis  of  administrative  factors  or  past  practice. 

Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Knowledge  of  the  effects  on  learning  of  different  sequencing  plans 
is  available. 

Procedures 

1.  Identify  commonality  of  subject  matter  and  anticipated  transfer  of 
learning  between  objectives. 

2.  Identify  degree  of  dependency  between  objectives. 

3.  Select  overall  sequencing  principle(s). 

4.  Group  and  sequence  objectives. 

Analysis  of  Service  Models 

ITRQ  Model.  Objectives  are  first  categorized  as  dependent,  independent,  or  sup¬ 
portive  (those  between  which  some  transfer  of  learning  occurs).  The  dependent  objectives 
are  then  sequenced  as  necessary  to  insure  that  prerequisite  learning  will  occur  first.  Next, 
supporting  objectives  are  placed  as  close  to  each  other  as  possible  without  interfering  with 
the  dependent  sequencing.  Independent  objectives  are  sequenced  last,  in  any  order  that 
is  practical.  The  next  step  is  to  identify  identical  objectives  and  objectives  with  identical 
actions.  Deletions  are  made  to  identical  objectives  so  that  each  is  taught  only  once. 
Objectives  with  identical  actions,  but  different  objects  of  the  action,  are  grouped  together. 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  procedures  are  identical  to  the 
ITRO,  except  that  no  mention  is  made  of  identical  or  identical-action  objectives. 

Air  Force  Pamphlet  50-58.  The  terminology  used  is  slightly  different  from  the 
ITRO/Marine  models,  but  provides  essentially  the  same  procedures. 

Commentary.  The  only  unambiguous  guidance  from  the  Service  models  is  to 
place  dependent  objectives  later  in  the  sequence  of  instruction  than  the  objectives  on 
which  they  depend.  Given  the  inadequacy  of  knowledge  about  how  different  sequencing 
strategies  affect  learning,  and  the  difficulty  of  estimating  transfer  of  learning,  this  may 
well  be  all  that  the  models  can  do,  leaving  the  rest  to  the  individual  judgment  of  the 
training  developer. 


Development  of  Plan  for 
Authoring  and  Managing  Instruction 

A  written  plan  must  be  made  available  to  those  who  will  author,  conduct,  and 
manage  the  instructional  program.  Whether  in  a  single  master  plan  or  in  separate  docu¬ 
ments  intended  for  the  different  personnel  concerned,  the  decisions  and  specifications 
made  in  previous  ISD  steps  must  be  transmitted  to  those  who  will  carry  them  out. 
Thus,  the  content  and  structure  planned  for  each  lesson  must  be  specified  to  those  who 
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will  author  instruction.  Information  about  the  presumed  liteiacy  level,  experience,  age, 
etc.,  of  the  target  audience  will  need  to  be  provided,  and  the  type  and  frequency  of 
testing  specified.  The  same  information  must  be  made  available  to  the  instructors  and 
course  managers.  In  addition,  they  must  be  informed  about  such  administrative  matters 
as  class  schedules,  instructor  allocations,  and  equipment  and  facilities  use.  All  of  this 
information  is  provided  in  a  plan(s)  for  developers  and  managers  of  instruction. 

Although  the  provision  of  a  plan  is  discussed  here  as  a  distinct  ISD  step,  the  content 
of  the  plan  so  far  as  lesson  specifications  is  concerned  has  been  almost  wholly  determined 
by  the  decisions  made  in  earlier  steps  which  specifies  the  grouping  and  sequencing  of 
objectives,  instructional  activities,  methods,  and  media.  If  the  earlier  decisions  and 
specifications  about  instructional  design  have  been  recorded  in  a  form  that  successfully 
conveys  this  information  to  those  who  will  use  it,  then  a  good  portion  of  lesson  specifi¬ 
cation  has  been  accomplished. 

What  remains  is  to  organize  the  course  into  its  units/lessons,  specify  the  objective(s) 
for  each  unit,  and  describe  the  instructional  plan  for  each  objective  (e.g.,  how  the  trainee 
is  to  be  informed  about  the  objective,  how  stimulus  material  is  to  be  presented  and  per¬ 
formance  elicited,  how  guidance  and  feedback  are  to  be  provided).  If,  however,  the 
decisions  made  earlier  have  not  been  recorded  in  usuable  form,  this  must  be  done  at 
this  point  as  part  of  the  lesson  specification  process,  prior  to  authoring  instruction. 

In  either  case,  decisions  must  be  made  about  administrative  matters  such  as  class 
schedules  and  equipment  use.  Although  such  information  may  not  be  necessary  to  develop 
instruction,  its  development  is  included  in  this  step  for  the  sake  of  completeness.  Actually, 
aspects  of  the  plan  that  have  no  bearing  on  lesson  specification  can  be  developed  either 
now  or  later,  so  long  as  the  plan  is  complete  before  instruction  is  implemented. 

Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Objectives,  instructional  activities,  methods,  and  media  have  all 
been  specified. 

Procedures 

1.  Specify  the  content  and  design  of  each  lesson. 

2.  Specify  how  the  instruction  will  be  conducted  and  managed. 

Analysis  of  Service  Models 

ITRO  Model.  The  ITRO  model  specifies  that  a  System  Master  Plan  be  developed 
to  indicate  how  the  course  will  be  conducted,  how  students  will  be  managed  and  tested, 
how  instruction  will  be  developed  and  evaluated,  and  other  elements.  Media  specifica¬ 
tions  are  made  separately. 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  model  specifies  that  a  Consoli¬ 
dated  Training  Data  Worksheet  be  constructed  for  each  objective,  specifying  instructional 
strategies  (methods),  instructional  activities,  logistical  considerations,  and  the  main  and 
supporting  ideas  for  the  instruction. 

Air  Force  Pamphlet  50-58.  The  AFP  50-58  model  specifies  that  the  instructional 
activities,  methods,  guidance  to  the  instructor,  estimated  instructional  hours,  equipment, 
and  media  required  be  specified  for  all  objectives  in  a  Program  of  Instruction  (POI). 

Commentary.  Each  of  the  models,  then,  requires  that  some  plan  or  outline  be 
provided.  As  mentioned,  this  step  may  be  combined  with  others— as  it  is,  for  example, 
in  the  ITRO  and  Marine  models.  The  adequacy  of  different  models  is  discussed  in  the 
four  preceding  ISD  steps  where  the  information  that  constitutes  the  lesson  specification 
is  developed. 


Review  and  Selection  of  Existing  Materials 

After  instructional  activities  and  media  have  been  specified,  and  before  development 
of  instruction  begins,  any  available  existing  instructional  materials  are  examined  to  see 
if  they  match  the  specifications  in  whole  or  part.  If  they  do,  such  materials  can  be 
utilized,  with  resultant  savings  in  the  next  ISD  step  of  authoring  instruction.  However, 
in  addition  to  satisfying  the  requirements  for  instructional  activities,  methods,  and  media, 
such  materials  must  also  be  appropriate  to  the  expected  trainee  characteristics  (experience, 
reading  ability,  education,  etc.)  and  in  harmony  with  the  instructional  management  plan 
of  the  proposed  course. 

The  distinguishing  characteristic  of  the  ISD  approach  in  this  step  is  that  the  suita¬ 
bility  of  existing  materials  is  judged  according  to  the  specifications  that  have  now  been 
established  for  instructional  activities,  methods,  and  media.  Although  these  specifications 
have  some  latitude  that  would  allow  changes  in  order  to  match  existing  materials,  this 
is  only  permissible  if  the  alternate  activities,  methods,  and  media  are  also  appropriate  to 
the  objectives. 

If  existing  materials  are  to  be  reviewed  for  their  appropriateness  to  the  newly 
established  specifications,  it  follows  that  this  review  cannot  precede  such  specification. 
Indeed,  if  existing  materials  are  examined  before  activities,  methods,  and  media  are  speci¬ 
fied,  it  is  all  too  easy  for  the  former  to  influence  the  latter.  The  ISD  approach  is  to 
determine  what  form  the  instruction  should  take  according  to  what  must  be  learned,  not 
according  to  what  materials  happen  to  be  available. 

Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Instructional  activities  have  been  specified. 

2.  Methods  of  training  have  been  specified. 

3.  Media  have  been  specified. 

Procedures 

1.  Examine  existing  instructional  materials  to  determine  whether  any  meet 
the  specifications  for  instructional  activities,  methods,  and  media. 

2.  Select  materials  or  parts  of  materials  that  meet  the  specifications,  or 
which  could  be  efficiently  revised. 

Analysis  of  Service  Models 

ITRO  Model.  After  instructional  activities  have  been  identified,  existing  materials 
are  reviewed  to  determine  whether  some  of  the  effort  of  developing  new  instructional 
materials  can  be  avoided.  This  review  is  to  extend  to  materials  produced  for  other  courses, 
in  other  Services,  and  in  the  civilian  educational  and  training  communities.  Whether 
existing  materials  can  be  used  in  the  course  under  development  is  determined  by  their 
appropriateness  to  the  specified  objectives,  trainee  characteristics,  learning  guidelines,1 
methods,  and  media.  Materials  that  are  nearly,  but  not  entirely,  adequate,  may  be 
revised.  Moreover,  some'  changes  may  be  made  in  the  specifications.  The  model  states 
that  the  specifications  for  learning  guidelines,  methods,  and  media  may  be  considered 
more  flexible  than  those  of  trainee  characteristics  or  the  objectives  themselves.  Any 
materials  selected  must  be  subjected  to  validation  in  a  later  step. 


'Learning  guidelines  are  used  in  specifying  instructional  activities,  see  p.  31. 
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Marine  Corps  Order  P1510.23B.  The  Marine  Corps  model  places  this  step  prior  to 
determining  methods  and  media.  The  adequacy  of  existing  materials  is  determined  by 
their  appropriateness  only  to  objectives,  trainee  characteristics,  instructional  activities, 
and  test  items.  (The  manual,  perhaps  by  oversight,  includes  “instructional  strategy”  as  a 
criterion.  Instructional  strategy,  i.e.,  methods  and  media,  is  determined  in  the  next  step.) 
Only  Marine  Corps  materials  are  considered  in  the  review.  There  is  no  mention  in  this 
step  of  subsequent  validation  of  selected  existing  materials,  nor  in  the  step  “Validate 
Instruction,”  where  the  only  instructional  input  listed  is  newly  developed  materials.  The 
placing  of  this  step  ahead  of  determining  instructional  strategy  would  save  the  effort 
involved  in  selecting  media  and  methods  in  those  cases  where  existing  materials,  using 
existing  media  and  methods,  were  judged  adequate.  However,  a  tendency  might  exist 
to  adopt  materials  based  on  instructional  strategies  that  were  actually  inappropriate,  if 
the  strategies  had  not  been  specified  beforehand  in  an  independent  step. 

Air  Force  Pamphlet  50-58.  This  model  does  not  discuss  the  review  and  adoption  of 
existing  materials  as  a  distinct  ISD  step.  This  would  not,  presumally,  prohibit  the  use 
of  existing  materials  in  a  course. 

Commentary.  Both  the  ITRO  and  Marine  Corps  models  make  clear  that  the  selec¬ 
tion  of  existing  materials  is  to  be  based  on  specified  characteristics  of  the  materials, 
rather  than  simply  on  personal  judgment.  In  considering  the  absence  of  this  step  in 
the  AFP  50-58  model,  it  should  be  noted  that  there  is  also  an  absence,  in  the  next  ISD 
step,  of  any  specific  guidance  for  authoring  instructional  materials.  Since  the  selection 
of  existing  materials  is  in  lieu  of  developing  new  ones,  the  model  is  consistent  when  it 
omits  both  steps. 

Authoring  of  Instruction 

After  existing  materials  have  been  reviewed,  and  suitable  ones  adopted  or  revised, 
new  instruction  is  developed  as  needed.  The  content  of  instruction  is  determined  by  the 
earlier  decisions  on  grouping  and  sequencing  of  objectives,  instructional  activities,  methods, 
and  media  and  by  the  specification  of  content  for  individual  units/lessons  provided  in  the 
authoring  plan.  The  actual  authoring  process  consists  of  such  activities  as  writing  scripts, 
and  preparing  tape/slide  presentations. 

ISD  models  are  not  intended  to  provide  specific  guidance  with  regard  to  the  many 
technical  skills  needed  in  authoring  instruction.  Rather,  the  step  is  included  in  the 
models  to  specify  its  relationship  to  earlier  and  later  steps  and  to  indicate  the  empirical 
trial-and -error  orientation  of  the  authoring  process. 

As  first  drafts  of  instructional  materials  are  produced,  small  portions  are  tried  out 
on  individuals  who  are  representative  of  the  entry  population.  The  materials  are  then 
revised  to  correct  weaknesses,  omissions,  and  ambiguities. 

A  distinguishing  characteristic  of  the  ISD  approach  to  authoring  materials  is  initially 
to  include  only  the  bare  minimum  of  instruction,  and  subsequently  to  augment  instruc¬ 
tion  as  needed.  This  approach  is  used  to  prevent  the  inclusion  of  extraneous  material, 
which  could  easily  be  incorporated  if  an  attempt  were  made  to  be  comprehensive. 

Since  instruction  is  deliberately  designed  to  be  just  adequate,  the  tryout  process  is  an 
essential  part  of  ISD.  Initial  drafts  are  not  expected  to  be  completely  satisfactory  and 
it  should  be  found  necessary  to  augment  instruction.  Only  through  this  progression 
can  the  economies  of  minimal  instruction  be  assured.  In  a  later  step,  quantitative  infor¬ 
mation  about  the  effectiveness  of  instruction  will  be  obtained  using  larger  groups  of 
trainees,  and  further  opportunity  for  revision  will  be  available. 


39 


Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Objectives  have  been  grouped  and  sequenced. 

2.  Instructional  activities  have  been  specified. 

3.  Methods  of  training  have  been  specified. 

4.  Media  have  been  specified. 

5.  Lesson  structure  and  content  have  been  planned. 

Procedures 

1.  Develop  lean  instruction. 

2.  Try  out  instruction  on  small  number  of  persons  representative  of  students. 

3.  Revise  and  augment  instruction  as  necessary. 

Analysis  of  Service  Models 

ITRO  Model.  The  ITRO  model  specifies  that  only  a  minimum  of  instruction  be 
included  in  first  draft  materials.  This  instruction  is  then  tried  out  on  one  student  at  a 
time,  and  revisions  made  as  needed.  The  ITRO  manual  provides  general  guidance  for 
writing  or  developing  many  different  types  of  materials:  audiovisual  scripts,  slide/tape 
programs,  television,  programmed  texts,  and  other  printed  materials,  platform  lectures, 
self -teaching  exportable  packages,  job  performance  aids,  and  so  forth. 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  model  also  specifies  that  only 
the  minimum  instruction  necessary  to  achieve  the  objective;  be  used.  In  addition,  the 
model  requires  that  a  plan  of  the  proposed  instruction  (“Concept  Sheet”)  be  approved 
prior  to  beginning  work.  No  specific  guidance  on  authoring  materials  themselves  is 
presented.  The  model  specifies  that,  if  possible,  instructional  materials  be  tried  out  on 
“a  single  representative  learner.” 

Air  Force  Pamphlet  50-58.  The  AFP  50-58  model  specifies  that  materials  be  tried 
out  on  a  small  number  of  students,  but  not  until  they  have  first  been  subjected  to  an 
internal  review  by  other  subject  matter  experts.  The  model  provides  guidance  on 
revising  materials  during  the  tryout  phase,  including  how  to  remedy  different  types  of 
failure  to  learn  (retention,  transfer,  acquisition). 

Commentary.  All  of  the  models,  then,  provide  for  a  tryout  of  materials  during  devel¬ 
opment,  and  the  ITRO  and  Marine  models  emphasize  that  instruction  should  be  lean. 

The  absence  of  specific  authoring  guidance  in  the  Marine  Corps  and  AFP  50-58  models 
should  not  be  considered  a  deficiency,  since  description  of  the  many  different  technical 
authoring  procedures  is  not  intended  to  be  included  in  ISD  manuals. 


Validation  of  Instruction 

After  instruction  has  been  developed  but  before  it  is  put  into  use,  it  must  be  tried 
out  to  see  if  it  works— that  is,  to  see  whether  trainees  attain  the  objectives.  This  is  done 
by  administering  the  course  (or  major  parts  of  the  course),  under  conditions  that  closely 
approximate  its  intended  use,  to  groups  of  trainees  representative  of  the  entry  population. 
Objective-referenced  achievement  tests  developed  prior  to  and  independent  of  instructional 
development  per  se  are  the  primary  validation  criteria.  Other  measures  of  instructional 
adequacy,  such  as  time  to  complete  lessons  and  acceptability  to  trainees,  may  also  be 
obtained  at  this  time.  Instruction  found  to  be  deficient  on  any  of  the  chosen  criteria  is 
revised  and  subjected  to  the  validation  process  again. 
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Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Objective-referenced  achievement  tests  are  available. 

Procedures 

1.  Specify  achievement  test  validation  criteria  (number  and  percent  of 
persons  in  validation  sample  required  to  pass  tests). 

2.  Specify  additional  validation  criteria. 

3.  Present  instruction,  administer  achievement  tests,  analyze  results,  revise 
instruction,  and  repeat  cycle  until  validation  criteria  are  met. 

Analysis  of  Service  Models 

ITRO  Model.  The  ITRO  procedure  is  to  conduct  both  individual  and  group 
validation  trials.  It  must  first  be  decided  what  types  of  information  will  be  needed  to 
measure  the  adequacy  of  the  materials  (e.g.,  achievement  tests,  student  attitudes,  time 
to  complete  lesson)  and  what  criterion  values  will  be  considered  adequate.  Instruments 
for  collecting  and  recording  the  data  must  then  be  developed. 

Before  trials  are  conducted,  entry  tests  and  pretests,  if  available,  are  used  to  select 
suitable  subjects.  In  individual  trials,  while  the  instruction  is  being  presented,  trainee 
questions  and  any  help  given  are  recorded.  When  the  trainee  has  completed  the  lesson, 
the  post-test  is  administered,  followed  by  attitude  and/or  other  chosen  measures.  Fol¬ 
lowing  individual  trials  with  from  three  to  six  trainees,  the  materials  are  revised  in  the 
light  of  test  results,  student  comments,  and  whatever  other  data  were  collected.  The  model 
acknowledges  that  a  good  deal  of  judgment  is  required  in  diagnosing  of  deficiencies  in 
the  materials.  When  it  is  felt  that  the  lessons  are  adequate,  group  trials  are  undertaken. 

The  model  provides  a  method  for  determining  the  sample  size  needed  to  obtain 
statistically  significant  results  in  group  trials,  as  a  function  of  different  standards  for 
mastering  objectives.  Possible  criteria  to  be  used  in  setting  such  standards  are  similar  to 
those  used  earlier  in  task  selection  (e.g.,  task  delay  tolerance,  availability  of  trained  incum¬ 
bents,  learning  difficulty).  The  model  cautions  against  being  too  rigid  in  setting  standards, 
however,  and  acknowledges  that  a  great  deal  of  error  is  present  in  any  attempt  to  measure 
performance.  If  the  trainees  fail  to  reach  the  criterion  specified,  the  “revision  cycle” 
(presumably  the  validation  procedure)  must  be  repeated.  The  ITRO  model  acknowledges 
that  it  may  not  be  possible  to  complete  validation  before  implementing  the  course,  and 
that  continual  revision  with  the  actual  trainees  may  be  required. 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  model  specifies  that  instructional 
materials  be  tried  with  progressively  larger  groups  and  revised  “until  they  are  satisfactory.” 
First,  the  materials  are  tested  on  a  single  representative  learner,  revised,  and  tried  out 
again  until  “consistent  errors  are  removed.”  Second,  group  trials  are  conducted  until 
“the  desired  mastery  criteria  have  been  obtained.”  The  manual  states  that  “group,  in 
this  case,  does  not  necessarily  mean  that  the  trials  must  be  conducted  with  a  group  of  a 
certain  size.”  The  Marine  Corps  model,  like  the  ITRO,  acknowledges  that  it  may  be 
necessary  to  validate  materials  by  continual  revision  with  actual  trainees. 

Air  Force  Pamphlet  50-58.  The  AFP  50-58  model  also  recommends  a  two-part 
validation  procedure.  In  the  first  part,  the  instruction  is  presented  to  a  group  of  six  to 
ten  students,  evenly  divided  between  low,  medium,  and  high  aptitude.  Information 
about  error  rates,  and  time  to  complete,  collected  during  this  tryout  is  then  used  as  the 
basis  for  revision.  Instruction  is  then  presented  to  another  group  of  six  to  ten  similar 
students.  The  cycle  of  instruction,  test,  and  revision  is  continued  “until  it  is  proven  that 
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the  students  can  perform  to  the  level  specified  in  the  criterion  objectives  and  tests,” 
or  until  at  least  20  students  have  been  through  the  instruction,  whichever  occurs  later. 

In  the  second  part,  complete  instructional  sequences  (ideally  the  entire  course  of  . 
instruction)  are  presented  to  a  group  of  about  30.  This  is  called  an  “operational  tryout,” 
and  is  conducted  using  the  equipment,  facilities,  and  administrative  procedures  that  will 
be  used  when  the  course  is  operational.  If  students  fail  to  meet  objectives,  the  model 
requires  the  designer  to  reassess  the  task  analysis,  objectives,  tests,  and  instruction  design 
and  development. 

Commentary.  The  procedures  recommended  in  the  different  models  are  essentially 
similar,  although  the  Marine  Corps  model  allows  considerably  more  leeway  in  the  size  of 
the  sample  for  group  trials.  Any  of  the  three  sets  of  procedures,  if  put  into  practice, 
would  be  adequate  to  determine  whether  the  training  objectives  had  been  achieved  as 
measured  by  the  achievement  tests. 


Internal  Evaluation 

After  the  instruction  has  been  implemented,  its  actual  effectiveness  must  be  deter- 
/  mined.  Although  validation  in  the  preceding  step  indicated  that  persons  who  received  the 
instruction  would  attain  the  objectives,  this  was  only  an  estimate,  based  on  an  approxi¬ 
mation  of  real  training.  How  well  the  instruction  works  during  actual  training  with 
actual  trainees  will  now  be  determined.  While  validation  occurred  at  one  time  only, 
internal  evaluation  will  be  continuous,  since  the  capabilities  of  trainees  and  the  manner 
in  which  instruction  is  conducted  may  change  over  time.  The  principal  measure  of 
instructional  effectiveness  is  the  trainees’  performance  on  the  objective-referenced 
achievement  tests.  Other  measures  include  trainees’  and  instructors’  opinions  and  atti¬ 
tudes  concerning  the  instruction,  time  to  complete  lessons,  and  attrition  rates.  This 
evaluation  of  instructional  effectiveness  is  sometimes  termed  product  evaluation. 

As  deficiencies  in  the  instruction  are  discovered,  an  attempt  is  made  to  identify 
their  causes  and  recommend  remedies.  In  locating  the  source  of  a  problem,  it  will  be 
important  to  know  what  ISD  steps  had  previously  been  performed,  what  decisions  were 
made,  the  rationales  for  these  decisions,  and  the  way  in  which  they  affected  training. 

It  would  be  pointless  to  consider  revising  the  choice  of  particular  instructional  activities, 
for  example,  without  first  considering  whether  and  how  they  had  actually  been  incorporated 
into  the  instruction.  This  analysis  or  evaluation  of  how  the  ISD  process  itself  was  carried 
out  is  also  part  of  internal  evaluation,  and  is  sometimes  termed  process  evaluation. 

Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Records  of  students’  performance  on  achievement  tests  are  available. 

2.  Documentation  is  available  of  what  occurred  during  the  ISD  process, 
including  such  elements  as  rationales  for  decisions,  and  departures 
from  standard  procedures. 

Procedures 

1.  Specify  evaluation  criteria  (number  and  percent  of  persons  required 
to  pass  tests,  etc.). 

2.  Identify  the  causes  of  shortcomings  in  the  instruction  and  specify 
revisions. 
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Analysis  of  Service  Models 

ITRO  Model.  The  ITRO  model  specifies  that  the  effectiveness  of  both  the  instruc¬ 
tion  and  the  ISD  effort  itself  be  evaluated.  Instructional  effectiveness  is  measured  by 
trainees’  attainment  of  the  objectives,  as  represented  by  test  scores.  The  methods  used 
are  basically  the  same  as  validation  procedures.  The  manual  also  recommends  collecting 
information  such  as  trainee  background,  entry  skills,  time  to  complete  lessons,  trainees’ 
evaluations  of  the  media  and  methods,  and  instructors’  evaluation  of  the  content.  The 
product  resulting  from  these  activities  is  a  report  recommending  revisions  to  improve  the 
course. 

The  other  aspect  of  internal  evaluation— progress  and  process  evaluation— is  actually 
begun  earlier.  It  begins  with  scheduling  the  ISD  steps  and  determining  which  parts  of 
each  step  will  apply  to  the  specific  project.  Then,  as  work  proceeds,  reports  are  made 
concerning  quality,  adherence  to  schedules,  departures  from  plans,  rationales  for  deci¬ 
sions,  and  other  relevant  factors.  At  the  end  of  each  step  a  report  is  made  as  to  whether 
the  products  of  that  step  are  good  enough  to  serve  as  input  to  ensuing  steps. 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  model  specifies  that  the  follow¬ 
ing  be  evaluated,  preferably  by  a  person  or  group  independent  of  course  design  and 
instructor  personnel:  (a)  whether  students  are  mastering  the  objectives,  (b)  whether 
the  course  was  developed  according  to  required  procedures  and  schedules,  and  (c)  whether 
the  course  is  being  administered  as  planned.  A  sample  form  is  provided  for  use  in  record¬ 
ing  trainee  test  performance.  In  the  sample  Evaluation  Plan,  it  is  recommended  that  when 
fewer  than  80%  of  the  trainees  are  mastering  80%  of  the  objectives  “the  instruction  will 
become  suspect.”  Checklists  are  to  be  used  to  collect  data  on  course  administration. 
Student  and  instructor  evaluations  of  the  instruction  are  to  be  obtained.  The  product 
of  this  step  is  a  set  of  recommendations  to  resolve  problems  and  weaknesses. 

Air  Force  Pamphlet  50-58.  A  quality  control  team,  free  from  the  influence  of  the 
instructional  staff,  is  to  conduct  the  internal  evaluation.  General  qualifications  for  team 
members  are  listed  in  the  manual.  Part  of  the  team’s  responsibility  is  to  use  the  follow¬ 
ing  procedure  to  evaluate  trainees’  mastery  of  objectives: 

1.  Select  a  random  sample  of  students  about  to  graduate,  administer  course 
criterion  tests,  and  report  the  results.  Before  testing,  the  team  decides 
what  will  be  the  passing  score. 

2.  Administer  the  corresponding  diagnostic  test  whenever  there  is  poor  per¬ 
formance  on  a  course  criterion  test,  to  isolate  the  cause  of  the  deficiency. 

3.  Scrutinize  the  part  of  the  course  where  the  deficiency  seems  to  lie,  and 
suggest  ways  to  correct  the  deficiency. 

4.  After  changes  have  been  made,  readminister  the  tests  to  graduates  of 
the  revised  course. 

The  model  does  not  specify  sample  sizes  for  the  test  program,  but  requires  that,  over  a 
period  of  several  graduating  classes,  each  test  be  given.  The  quality  control  team  is 
also  responsible  for  examining  and  evaluating  compliance  with  course  control  documents, 
performance  of  instructors,  and  the  adequacy  and  appropriateness  of  training  aids  and 
equipment,  supplies,  and  facilities. 

Commentary.  All  of  the  models  require  that  trainees’  mastery  of  the  objectives  be 
used  as  a  criterion  for  determining  the  effectiveness  of  the  course.  Any  of  the  three 
sets  of  procedures,  if  put  into  practice,  would  be  adequate  to  measure  whether  the 
training  objectives  had  been  achieved.  The  Marine  Corps  model,  however,  provides  pri¬ 
marily  an  outline  of  the  step  rather  than  a  detailed  specification  of  procedures. 
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In  addition  to  determining  whether  trainees  are  attaining  course  objectives  (internal 
evaluation),  their  proficiency  and  the  adequacy  of  the  instructional  design  and  develop¬ 
ment  process  must  be  evaluated  by  a  standard  external  to  the  course:  the  performance 
of  the  graduate  on  the  job.  If  graduates  are  unable  to  perform  certain  tasks  when  they 
reach  the  job,  and  these  deficiencies  are  unacceptable,  the  course  may  have  to  be  revised. 
The  purpose  of  external  evaluation  is  to  discover  any  such  deficiencies,  identify  their 
causes  (e.g.,  improper  job  analysis,  inadequate  training),  and  recommend  remedies.  If, 
for  example,  the  job  has  changed  since  the  original  analysis,  or  if  the  analysis  was  faulty, 
the  list  of  tasks  on  which  the  course  is  based  may  have  to  be  changed.  If  the  training 
product  is  not  adequate,  instructional  activities  or  the  objectives  themselves  may  require 
revision.  If  skills  and  knowledge  have  been  forgotten  by  the  time  they  are  needed,  the 
remedy  may  be  to  increase  training,  train  on  the  job,  or  shorten  the  time  between  train¬ 
ing  and  performance  in  some  other  way. 

Probably  the  most  accurate  methods  of  external  evaluation  are  direct  observation 
and  testing  of  graduates  on  the  job.  Such  approaches  are  costly,  however,  so  some 
reliance,  if  not  all,  must  be  placed  on  supervisors’  summary  evaluations  (ratings)  of  per¬ 
formance.  Graduates’  evaluation  of  their  own  and  peers’  performance  may  be  included. 
Information  also  may  be  obtained  on  such  factors  as  what  tasks  are  performed,  what 
aspects  of  training  are  perceived  as  insufficient,  and  what  training  is  not  used. 

Whatever  the  source  and  type  of  evaluation  information,  it  should  be  obtained  at 
a  task  level  of  specificity.  More  general  evaluations  are  of  little  use  in  isolating  the 
causes  of  inadequate  performance.  In  all  cases,  the  external  evaluation  must  take  place 
fairly  soon  after  the  graduate  has  reached  the  field  (usually  within  a  few  months).  Other¬ 
wise,  it  will  be  difficult  to  discriminate  between  skills  and  knowledge  acquired  in  training 
and  those  acquired  on  the  job. 

Summary  of  Requirements 

Prerequisite  Conditions  and  Information 

1.  Access  to  supervisors  and  job  incumbents  is  possible  soon  after  arrival 
of  graduates  on  the  job. 

Procedures 

1.  Construct  evaluation  instruments  (mail  questionnaires,  job  sample  tests, 
interview  guides,  etc). 

2.  Collect  evaluation  information. 

3.  Analyze  data,  identify  causes  of  deficiencies,  and  specify  revisions. 

Analysis  of  Service  Models 

ITRO  Model.  Two  questions  are  considered  in  external  evaluation:  Can  the  graduate 
perform  their  job  tasks,  and  is  the  job  the  same  as  it  was  when  originally  analyzed?  The 
recommended  method  of  determining  whether  graduates  can  perform  their  tasks  is  to 
administer  Job  Performance  Measures  in  the  field  30-90  days  after  graduation.  Basing 
decisions  on  data  gathered  by  other  means  (questionnaire,  interview,  etc.)  is  described 
as  far  riskier. 

Determining  whether  the  original  job  analysis  is  still  valid  is  seen  as  a  matter  of 
determining  the  present  relationship  between  the  Job  Performance  Measures  and  actual 
job  requirements— that  is,  the  predictive  validity  of  the  Job  Performance  Measures.  Per¬ 
formance  on  the  Job  Performance  Measures  is  compared  with  supervisors’  evaluations  of 
graduates’  performance. 
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For  interpreting  the  information  from  interviews  and  questionnaires,  and  deciding 
whether  the  course  needs  revision,  the  model  acknowledges  that  no  hard  and  fast  rules 
apply.  It  provides  guidance,  however,  on  such  questions  as  how  to  interpret  data  when 
a  conflict  exists  between  supervisors’  ratings  and  Job  Performance  Measure  results.  The 
model  cautions  against  revising  simply  for  the  sake  of  revision,  and  recommends  making 
few,  if  any,  changes  if  the  majority  of  graduates  and  supervisors  are  satisfied  with  the 
quality  of  the  training. 

Marine  Corps  Order  P1510.23B.  The  Marine  Corps  model  states  that  the  purposes 
of  external  evaluation  are  to  determine  whether  the  graduates  are  performing  as 
trained  and  whether  the  job  has  changed  since  front-end  analysis.  The  principal  method 
is  to  survey  graduates  and  supervisors,  by  questionnaire  or  interview,  on  how  well  they 
believe  the  graduates  perform  the  job,  the  type  and  extent  of  training  received  on  the 
job,  the  effectiveness  and  relevance  of  instruction,  how  graduates  of  the  new  course  com¬ 
pare  to  graduates  of  earlier  training,  and  similar  points.  In  addition,  the  manual  states 
that  “the  validators  will  attempt  to  determine  how  well  the  graduates  scored  on  the  Job 
Performance  Measures  and  which  JPMs  gave  them  the  most  trouble.” 

Air  Force  Pamphlet  50-58.  The  AFP  50-58  model  specifies  that  external  evaluation 
may  be  accomplished  by  questionnaires,  observation,  interview,  and  formal  supervisor’s 
evaluation  during  the  graduates’  first  two  months  on  the  job.  Content  of  the  evaluation 
instruments  is  to  be  based  on  the  Job  Performance  Requirements  and  Training  Require¬ 
ments  listed  on  the  Training  Data  Worksheet.  Examining  the  data  collected  from  valida¬ 
tion  and  internal  and  external  evaluation  is  expected  to  show  whether  the  graduates  satisfy 
job  performance  requirements. 

Commentary.  The  ITRO  method,  administering  Job  Performance  Measures  to  grad¬ 
uates  in  the  field,  appears  to  be  the  most  reliable  way  to  measure  the  adequacy  of  the 
instructional  design,  but  also  the  most  costly.  The  ITRO  model  provides  the  most  guid¬ 
ance  for  isolating  the  causes  of  performance  discrepancies  revealed  during  external  eval¬ 
uation.  If  summary  evaluations  are  to  be  used,  the  ITRO  and  AFP  50-58  model  both 
specify  that  information  be  obtained  at  the  task  level  of  specificity,  while  the  Marine 
Corps  model  does  not.  The  Marine  Corps  model  is  unique  among  the  three  in  specifying 
a  delay  of  six  months,  rather  than  two  or  three,  before  surveying  graduates.  A  delay  of 
this  length  might  reduce  the  meaningfulness  of  the  findings,  since  much  of  an  incumbent’s 
skill  and  knowledge  at  that  point  may  have  been  acquired  on  the  job,  not  in  training. 


SUMMATION 

The  adequacy  of  ISD  and  the  Service  guidance  in  general  varies  according  to  the 
kind  of  ISD  step  in  question.  Three  general  types  of  steps  can  be  distinguished. 

1.  Procedural  steps.  Most  ISD  steps  involve  the  direct  application  of  procedures. 
The  means  for  carrying  them  out  are  generally  available.  Examples  are  Identification  of 
Job  Requirements,  Analysis  of  Tasks,  Identification  of  Entry  Behavior,  Development  of 
Achievement  Tests,  Validation  of  Instruction.  The  adequacy  of  ISD  with  respect  to 
procedural  steps  becomes  a  matter  of  the  clarity  and  completeness  of  the  guidance. 

Both  the  ITRO  and  AFP  50-58  models  (with  the  exceptions  noted  in  this  chapter) 
appear  adequate  in  this  regard.  The  Marine  Corps  model  is  in  the  nature  of  an  outline 
of  what  steps  must  be  accomplished  than  a  set  of  instructions  for  accomplishing  them, 
and  while  it  cites  the  ITRO  model  as  a  reference,  there  is  every  indication  that  it  is 
intended  to  be  sufficient  by  itself  for  carrying  out  ISD.  The  highly  abbreviated  form  of 
the  guidance  makes  its  intent  harder  to  discern,  thereby  presenting  wider  latitude  for 
misinterpretation  or  misapplication. 
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2.  Decisions  affecting  total  system  performance.  These  design  steps  involve  deci¬ 
sions  that  interface  with  other  components  of  the  operational  system  and  affect  force 
effectiveness  and  productivity.  The  steps  are  Selection  of  Tasks  for  Training  and  Selection 
of  Instructional  Setting.  Tasks  selected  for  training  establish  the  goals  of  training  and 
thereby  determine  the  output  of  the  training  subsystem.  Thus,  tasks  must  be  selected  on 
the  basis  of  their  contribution  to  overall  system  performance.  The  setting  that  is  selected 
for  training  interfaces  with  other  components  of  the  system  (e.g.,  it  may  reduce  the  pro¬ 
ductivity  of  supervisors  who  must  devote  time  to  on-the-job  instruction)  and  should  also 
be  selected  on  the  basis  of  its  system  effects. 

To  develop  information  about  the  relationship  between  system  performance  and 
criteria  for  selecting  tasks  and  setting  requires  a  criterion  of  system  performance  against 
which  variations  in  task  and  setting  can  be  tested.  Currently  available  criteria  of  system 
performance  such  as  those  used  to  assess  force  readiness  (e.g.,  personnel  and  equipment 
fill  vs.  authorization,  deadline  rate,  hours  on  equipment,  training  status)  are  either  insen¬ 
sitive  to  variations  in  training  or  not  reliable.  Because  the  effects  of  task  and  setting 
selection  cannot  be  satisfactorily  measured,  ISD  in  general,  and  as  represented  by  the 
Service  models  in  particular,  does  not  provide  an  adequate  methodology  for  making  deci¬ 
sions  that  affect  total  system  performance.  This  fact  is  reflected  in  the  lack  of  systematic 
procedures  in  the  Service  models  for  selecting  tasks  and  settings. 

3.  Decisions  affecting  training  system  performance.  These  design  steps  involve  deci¬ 
sions  that  affect  the  efficiency  of  training.  The  steps  are  Selection  of  Instructional 
Activities,  Selection  of  Instructional  Methods,  Selection  of  Media,  and  Grouping  and 
Sequencing  of  Instruction.  Once  the  goals  of  training  have  been  decided,  decisions  made 
in  these  steps  determine  the  efficiency  with  which  these  goals  are  attained.  The  present 
state  of  the  art  of  instructional  design,  however,  provides  only  incomplete  information 
for  making  these  decisions. 

Current  guidelines  for  selecting  instructional  activities,  methods,  media,  and  sequence 
are  rudimentary.  They  provide  only  the  starting  point  for  a  trial-and-error  approach  to 
maximizing  efficiency,  which— given  the  number  of  possible  combinations  of  methods— 
could  not  be  expected  to  succeed.  Thus,  while  ISD  does  provide  a  framework  for  com¬ 
paring  alternate  strategies  (if  time  and  resources  permit),  it  cannot  be  regarded  as  a  meth¬ 
odology  for  achieving  optimal  training  efficiency,  given  the  current  state  of  instructional 
technology. 
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Chapter  3 

A  MAIL  SURVEY  OF  ISD  APPLICATIONS 


At  the  initiation  of  this  study,  staff  members  visited  headquarters  organizations  in 
each  Service  to  identify  the  locations  of  applications  of  Instructional  System  Develop¬ 
ment  and  to  determine  lines  of  authority  and  responsibility  for  ISD  implementation.1 
Information  obtained  suggested  the  possible  use  of  ISD  in  many  organizations,  far  more 
than  could  be  individually  examined  in  the  study. 

In  order  to  identify  the  organizations  that  were  making  maximal  use  of  ISD  pro¬ 
cedures,  an  ISD  Activities  Questionnaire  (see  Appendix  A)  requesting  descriptions  of 
instructional  development  procedures  was  distributed  to  schools  and  units  in  each  Service 
that  were  engaged  in  training  and  training  development.  The  questionnaire  was  sent  to 
Army  schools  by  Training  Developments  Institute,  Army  Training  and  Doctrine  Command; 
to  Navy  fleet  training  centers,  technical  training  centers  and  training  detachments  by 
Chief  of  Naval  Education  and  Training;  to  Navy  air  crew  training  units  by  Headquarters, 
Naval  Air  Systems  Command;  to  Marine  Corps  schools  by  Headquarters,  Marine  Corps; 
and  to  Air  Force  MAJCOMS  by  the  Directorate  of  Operations  and  Readiness,  Headquarters, 
Air  Force. 

Because  in  some  instances  major  commands  reproduced  the  questionnaire  and  redis¬ 
tributed  copies  to  subordinate  units,  it  is  not  possible  to  specify  exactly  how  many  ISD 
Activities  Questionnaires  were  distributed.  The  questionnaires  were  returned  from  a 
total  of  209  organizations  in  the  four  Services. 

In  completing  the  questionnaire,  respondents  first  listed  all  courses,  either  existing 
or  under  development,  for  which  job  analysis  data  had  been  compiled.  Although  we  did 


1  Visits  were  made  to: 

Army 

Training  Developments  Institute, 
Training  and  Doctrine  Command 
Training  Support  Center, 

Training  and  Doctrine  Command 


Navy 

Chief  of  Naval  Operations 
Director  of  Naval  Education 
and  Training 

Chief  of  Naval  Education  and 
Training 

Naval  Air  Systems  Command 
Chief  of  Naval  Technical  Training 
Naval  Training  Equipment  Center 
Training  Analysis  and  Evaluation  Group 
Navy  Personnel  Research  and  Develop¬ 
ment  Center 


Marine  Corps 

Headquarters,  Marine  Corps 
Air  Force 

Directorate  of  Operations  and  Readiness 
Directorate  of  Personnel  Programs 
Air  Training  Command 
4444  Operations  Squadron  (ISD)  TAC 
Air  Force  Office  of  Scientific  Research 
Technical  Training  Division,  Human 
Resources  Laboratory 
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not  define  ISD  in  terms  of  any  specific  type  or  sequence  of  activity,  we  did  require  that 
job  analysis  data  be  available  for  developmental  efforts  entered  into  the  questionnaire. 

This  was  done  to  eliminate  from  the  study  instances  in  which  ISD  was  considered  to  be 
nothing  more  than  the  application  of  one  or  more  design  or  instructional  techniques  with¬ 
out  the  need  for  deriving  training  from  job  requirements.1  Thus  the  sample  was  restricted 
to  those  efforts  which  treated  the  derivation  of  training  from  specified  job  requirements 
as  a  fundamental  characteristic  of  ISD. 

Across  all  Services  a  total  of  1,814  training  courses  so  defined  were  identified 
(Table  1). 

Table  1 

Training  Courses  With  Job  Analysis 
Data  Available 


Service 

I  Organizations 

Responding 

Courses 

Army 

16 

357 

Navy 

65 

459 

Marine  Corps 

10 

75 

Air  Force 

118 

923 

Total 

209 

1814 

Of  the  209  organizations  responding,  205  (98%)  also  provided  information  on  the 
total  number  of  courses  they  conduct,  which  is  5,934.  Thus,  job  analysis  data  were 
reported  to  have  been  compiled  and  available  for  about  31-percent.  Of  these,  job  analysis 
data  were  reported  to  have  been  compiled  and  available  for  1,648  (28%). 

For  the  courses  thus  listed,  a  check  list  of  20  potential  ISD  activities  was  provided 
to  identify  the  events  that  had  occurred  in  course  development  (Table  2).  Respondents 
indicated  whether  each  potential  activity  had  been  undertaken,  whether  it  had  been 
completed,  ard  whether  products  of  the  activity  were  currently  available. 

The  questionnaire  survey  was  intended  primarily  to  provide  information  for  selecting 
development  efforts  to  be  examined  during  subsequent  visits.  The  data  it  generated  are 
of  limited  value  for  analytic  purposes  since  (a)  respondents  differed  in  their  interpre¬ 
tation  of  the  items  in  the  checklist,  and  (b)  the  activities  reported  were  later  found  to 
be  exaggerated  when  checked  in  the  organizations  visited.  However,  the  data  are  of 
some  interest  since  they  reflect  to  a  certain  degree  the  general  pattern  of  ISD  applications 
that  was  subsequently  revealed  in  the  field  visits. 

Percentages  of  responses  for  each  ISD  step  conducted  for  each  Service,  and  for  all 
Services  combined,  are  given  in  Tables  3-7.  The  greatest  number  of  ISD  steps  performed 


’Where  training  for  specific  jobs  is  being  developed,  requirements  most  often  take  the  form  of  a 
listing  of  tasks  to  be  performed.  Where  instruction  is  designed  to  meet  educational  goals,  development 
may  begin  with  a  specification  of  more  general  capabilities  (skill,  knowledge)  that  are  recognized  to 
support  many  different  activities.  An  explicit  statement  of  either  tasks  to  be  performed  or  capabilities 
to  be  acquired  is  necessary  for  an  objective  determination  of  instructional  requirements. 
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Table  2 


Checklist  of  Potential  Activities  in 
ISD  Activities  Questionnaire 

1 .  Analyzed  probelm/system/existing  course  to  determine  need  for  training  development. 

2.  Obtained  priority  information  about  tasks  (frequency,  delay  tolerance,  criticality,  etc.) 

3.  Analyzed  tasks  into  elements,  conditions,  and  standards. 

4.  Selected  tasks  for  training  on  the  basis  of  task  priority  information. 

5.  Clustered  tasks  and  selected  instructional  setting  on  the  basis  of  priority  information, 
resources,  and  output  requirements. 

6.  Developed  terminal  and  intermediate  learning  objectives. 

7.  Sequenced  and  clustered  objectives. 

8.  Developed,  tried  out,  and  revised  tests. 

9.  Identified  knowledge  and  skills  of  entering  trainees. 

10.  Identified  type(s)  of  learning  required  for  each  objective,  and  specified  corresponding 
learning  activities. 

11.  Identified  media  appropriate  to  type  of  (earning  and  learning  activities. 

12.  Specified  plan  for  pacing,  instructor  role,  group/individualized  presentation, 
scheduling,  entry/exit  requirements,  etc. 

13.  Reviewed/selected  appropriate  existing  instruction  materials.3 

14.  Authored/produced  new  instruction  materials. 

15.  Validated  instruction  materials. 

16.  Conducted  instruction. 

17.  Analyzed  student  performance  (for  internal  evaluation  of  course). 

18.  Revised  training  according  to  results  of  internal  evaluation. 

19.  Determined  if  graduates  were  meeting  performance  requirements  on  the  job 
(external/field  evaluation). 

20.  Revised  training  according  to  results  of  external  evaluation. 


aThi*  activity  was  inadvertently  omitted  from  the  questionnaire  sent  to  Army  organizations. 
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Table  4 

Percent  of  Responses  for  Each  ISD  Step  in 
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Table  7 

Percent  of  Responses  for  Each  ISD  Step  in  1814 
Instructional  Development  Efforts:  All  Services  Combined 
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(i.e.,  highest  mean  percentage  of  responses  for  all  potential  activities  in  the  category 
“activities  undertaken/products  available”)  is  reported  by  the  Marine  Corps  (73%),  fol¬ 
lowed  by  the  Air  Force  (63%),  the  Navy  (49%),  and  the  Army  (35%). 

The  relatively  low  overall  incidence  of  ISD  activity  reported  by  the  Army  is  a 
consequence  of  their  current  focus  on  developing  exportable  training  and  evaluation 
materials  for  use  in  operational  units  (e.g.,  Soldier’s  Manuals,  Training  Extension  Course 
lessons,  and  Skill  Qualification  Tests),  rather  than  instructional  development  per  se  (see 
Chapter  4).  This  is  seen  more  clearly  when  the  percentages  of  responses  for  different 
types  of  ISD  activities  are  examined.  Three  major  phases  of  ISD  are  (a)  analysis  of  job 
and  training  requirements,  (b)  development  of  instruction,  and  (c)  evaluation  of  instruc¬ 
tion.  Items  from  the  activities  checklist  that  most  clearly  represent  each  of  these  phases 
were  selected.  The  mean  response  percentages  in  the  category  “activities  undertaken/ 
products  available”  for  each  phase  by  Service  are  shown  in  Table  8. 


Table  8 

Level  of  Activity  in  Three  Phases  of  ISD 


Mean  Percent  of  Responses  Indicating 

Activity  and  Products 

Service 

Analysis8 

Development13 

Evaluation0 

Army 

76.6 

30.6 

9.4 

Navy 

57.9 

60.4 

40.0 

Marine  Corps  , 

50.7 

82.6 

72.9 

Air  Force 

63.0 

65.5 

56.2 

All  Services  Combined 

63.9 

59.9 

44.2 

aBased  on  items:  Obtained  priority  information  about  tasks  (frequency,  delay 
tolerance,  criticality,  etc.);  analyzed  tasks  into  elements,  conditions,  and  standards;  and 
selected  tasks  for  training  on  the  basis  of  task  priority  information. 

bBased  on  ite  -is:  Developed  terminal  and  intermediate  learning  objectives;  sequenced 
and  clustered  objectives;  identified  type(s)  of  learning  required  for  each  objective  and 
specified  corresponding  learning  activities;  and  identified  media  appropriate  to  type  of  learning 
and  learning  activities. 

c8ased  on  items:  Validated  instruction  materials;  analyzed  student  performance  (for 
internal  evaluation  of  course);  and  determined  if  graduates  were  meeting  performance  require¬ 
ments  on  the  job  (external/field  evaluation). 


In  Army  applications,  only  35  percent  (Table  3)  of  the  responses  for  all  types  of 
activities  indicated  actions  and  products,  whereas  activity  is  shown  in  77  percent  (Table  8) 
of  the  steps  concerned  with  analysis  of  job  and  training  requirements.  Front-end  analysis 
is  necessary  in  the  development  of  Soldiers  Manuals  and  prescribed  for  the  construction 
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of  Skill  Qualification  Tests.1  The  pattern  reported  by  Army  organizations  clearly  reflects 
these  activities. 

The  other  noteworthy  aspect  of  these  data  is  the  relatively  low  level  of  activity  in 
evaluation.  With  the  exception  of  the  Marine  Corps  (with  data  representing  a  smaller 
number  of  applications  and  a  far  smaller  number  of  organizations),  all  Services  report 
a  lower  level  of  activity  in  evaluation  than  in  either  analysis  or  development.  Also  less 
activity  is  reported  by  the  Army,  Navy,  and  Air  Force  for  any  of  the  three  subclasses 
of  evaluation  (validation,  internal  evaluation,  external  evaluation)  than  for  analysis  or 
development  (Table  9).  External  evaluation  was  reported  least  frequently. 

Table  9 

Level  of  Activity  in  Three  Types  of  ISD  Evaluation 


Percent  of  Responses  Indicating 
Activity  and  Products 


Service 

Validation 

Internal 

Evaluation 

External 

Evaluation 

Army 

15.7 

14.0 

7.8 

Navy 

42.0 

47.0 

30.9 

Marine  Corps 

69.4 

78.7 

70.7 

Air  Force 

55.1 

62.9 

50.7 

All  Services  Combined 

44.6 

50.0 

38.1 

'William  C.  Osborn,  Roy  C.  Campbell,  and  J.  Patrick  Ford.  Handbook  for  Development  of  Skill 
Qualification  Teats,  HumRRO  Final  Report  77-1,  January,  1977. 

Individual  Training  and  Evaluation  Directorate,  U.S.  Army  Training  Support  Center.  Guidelines  for 
Development  of  Skill  Qualification  Tests,  December  1977. 


Chapter  4 

SERVICE  METHODS  OF  ORGANIZING  FOR  ISD 


The  Services  differ  widely,  both  among  and  within  themselves,  in  the  ways  in  which 
they  have  organized  to  meet  the  requirements  of  Instructional  System  Development.  In 
part,  their  responses  reflect  different  notions  of  what  the  ISD  requirements  are,  and  in  part 
simply  the  different  situations  faced  by  the  four  Services.  At  the  most  universal  level, 
there  is  recognition  of  the  need  for  information  about  the  ISD  process  and  for  expertise 
to  carry  out  its  individual  steps.  One  response  to  this  need  has  been  to  revise  instructor 
training  courses  to  concentrate  on  skills  and  knowledge  related  to  ISD  rather  than  on  the 
traditional  content,  methods  of  effective  instructional  presentation.  Another  response 
has  been  to  specialize,  partitioning  the  ISD  process  into  several  phases  and  assigning  per¬ 
sonnel  to  develop  expertise  and  to  work  on  only  one  phase.  Another  approach  has  been 
to  assign  personnel  permanently  to  ISD  duties,  with  or  without  specialization  in  a  single 
phase.  Expertise  has  also  been  brought  to  bear  by  engaging  civilian  contractors  to 
accomplish  ISD  or  to  participate  in  joint  military /contractor  development  efforts. 

The  need  for  expertise,  or  the  lack  of  it,  is  also  reflected  in  the  degree  to  which 
ISD  activity  has  been  centralized.  One  view  is  that  expertise  can  be  acquired  by  instruc¬ 
tors  at  all  training  sites,  and  that  they  should  be  required  to  apply  ISD  as  they  develop 
and  revise  their  own  courses.  At  the  other  extreme  is  the  view  that  expertise  is  both 
limited  and  hard  to  acquire.  This  has  led  to  the  deliberate  placement  of  personnel  at  a 
limited  number  of  sites,  to  develop  training  to  be  conducted  at  other  locations.  This 
permits  the  use  of  personnel  who  already  possess  backgrounds  in  training  design  or  devel¬ 
opment,  and  who  can  be  expected  to  acquire  further  expertise  through  a  succession  of 
ISD  efforts.  Centralizing  ISD  activity  in  this  way,  however,  means  that  instructors  must 
be  willing  to  accept  training  material  they  did  not  develop  themselves.  Resistance  to 
doing  so  is  widely  acknowledged,  and  is  sometimes  termed,  aptly,  the  “not-invented-here 
syndrome.” 

In  addition  to  the  need  for  expertise,  there  is  some  belief  in  the  desirability  of 
performing  various  steps  separately  in  order  to  maintain  the  high  degree  of  objectivity 
implicit  in  ISD.  To  keep  the  determination  of  job  requirements,  for  example,  insulated 
from  the  influence  of  either  what  has  previously  been  taught  or  what  certain  persons 
believe  should  be  taught,  some  training  personnel  believe  that  job  task  lists  should  be 
developed  by  persons  other  than  instructors.  Another  example  is  the  obvious  appropri¬ 
ateness  of  assigning  the  responsibility  for  evaluating  training  to  others  than  those  who 
develop  it. 

To  separate  the  ISD  functions  requires,  of  course,  that  each  party  be  willing  to  use 
the  products  developed  by  others.  This  situation  is  analogous  to  that  of  centralized 
instructional  development,  and  the  degree  of  resistance  may  be  just  as  high  between  job 
analysts  and  training  analysts,  for  example,  as  between  developers  of  training  and 
instructors. 

Another  difficulty  inherent  in  separating  ISD  functions  is  achieving  the  appropriate 
degree  of  communication  between  those  performing  separate  steps.  Too  much  commu¬ 
nication  threatens  to  compromise  objectivity,  but  some  is  necessary  both  because  the 
steps  are  so  interdependent,  and  because  subject  matter  experts  (often  the  course  instruc¬ 
tors)  must  provide  input  at  several  points. 
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The  manner  in  which  the  individual  Services  have  organized  to  apply  ISD  is  sum¬ 
marized  in  the  following  sections. 


ARMY 

Training  is  developed  at  the  TRADOC  resident  training  center  proponent  for  a 
particular  MOS.  TRADOC  has  organized  its  schools  along  the  lines  of  the  ITRO 
model  for  ISD,  with  separate  directorates  or  divisions  for  different  ISD  functions.  Thus, 
for  example,  a  task  list  developed  in  a  Training  Analysis  Division  of  a  Directorate  of 
Training  Developments  would  be  handed  off  to  a  Course  Development  Division  for 
the  development  of  training,  and  the  resulting  training  would  be  evaluated  by  the 
Directorate  of  Evaluation.  (The  exact  arrangement  varies  from  school  to  school.)  The 
responsibility  of  each  division,  however,  is  primarily  the  generation  of  one  or  more 
products  for  the  Enlisted  Personnel  Management  System  (EPMS)  for  use  outside  the 
school.  These  include  lists  of  training  tasks  for  each  MOS  (Soldier’s  Manuals  and 
Commander’s  Manuals)  and  evaluation  instruments  and  training  materials  based  on  these 
lists  (Skill  Qualification  Tests,  Army  Training  and  Evaluation  Program  tests.  Training 
Extension  Course  lessons). 

Resident  training,  not  being  one  of  these  EPMS  products,  is  not  necessarily  based 
on  the  same  task  lists.  It  is  generally  developed  by  instructors  in  training  departments, 
independently  of  the  analysis  and  design  activities  of  the  various  divisions  in  the  school. 
The  activities  of  these  divisions,  then,  do  not  usually  lead  to  the  systematic  revision 
of  training,  at  least  initial  training,  for  an  MOS.  When  ISD  is  applied  to  resident  train¬ 
ing,  it  typically  follows  a  decision  to  convert  a  course  to  self-pacing.  In  such  cases, 
an  ad  hoc  team  is  sometimes  assembled  to  carry  out  the  process. 


NAVY 

For  initial  “A”  School  technical  training,  the  Navy  has  centralized  ISD  activity 
at  Instructional  Program  Development  Centers  (IPDC),  under  the  Chief  of  Naval 
Education  and  Training  Support.  Two  centers  have  been  established;  more  are 
planned.  One  course  developed  at  IPDC  San  Diego  has  already  been  turned  over  to 
Chief  of  Naval  Technical  Training  (CNTT)  for  implementation. 

To  the  extent  that  the  IPDC  approach  is  pursued,  the  responsibility  for  design¬ 
ing  and  developing  “A”  School  courses  will  be  removed  from  the  schools,  which  will 
be  responsible  only  for  conducting  the  training.  This  arrangement  has  ostensibly  been 
chosen  because  training  technologists  needed  to  support  ISD  cannot  be  made  available 
everywhere  training  is  conducted.  Centralization  of  training  development  also  pre¬ 
sumably  permits  more  effective  control  of  the  ISD  process.  Whether  such  control 
is  achieved  remains  to  be  seen.  The  IPDCs  are  staffed  primarily  by  civilian  education 
specialists  and  instructional  materials  developers.  Subject  matter  expertise  is  provided 
by  the  instructors  of  the  existing  courses,  who  also,  to  a  large  extent,  determine  what 
tasks  should  be  trained. 

Since  the  IPDC  approach  is  only  beginning  to  be  implemented,  most  Navy  “A” 
School  courses  are  still  developed  and  revised  at  the  technical  training  center  where 
they  are  taught,  by  course  instructors  and  civilian  education  or  training  specialists.  A 
Curriculum  and  Instructional  Standards  Office  is  typically  responsible  for  coordinating 
approval  of  the  course  by  CNTT.  More  advanced  training  at  Naval  Air  Maintenance 
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Training  Detachments  and  other  training  centers  under  CNTT  is  also  locally  developed 
and  subject  to  CNTT  approval.  An  analogous  situation  exists  for  training  conducted 
by  the  Atlantic  and  Pacific  Fleet  Training  Commands. 

Another  area  of  major  ISD  effort  in  the  Navy  is  aircrew  training.  The  Naval 
Training  Equipment  Center  (NTEC)  has  sponsored  several  large-scale  analysis  and 
development  projects  by  civilian  contractors,  for  new  or  updated  aircraft  (one  aircrew 
training  squadron  has  also  undertaken  its  own  effort,  patterned  after  the  contractor- 
developed  procedures).  In  some  cases  the  projects  have  been  limited  to  analysis  and 
design  of  training.  When  instructional  materials  have  been  developed  as  well,  the 
emphasis  has  been  on  self-paced  audiovisual  and  printed  media  to  replace  much  of 
the  instruction  that  was  formerly  conducted  by  a  one-to-one  tutorial  or  in  the  group- 
paced  lecture/demonstration  mode.  NTEC  has  also  developed  specifications  to  guide 
and  control  future  ISD  efforts  conducted  by  civilian  contractors.1 


MARINE  CORPS 

The  Marine  Corps  has  not  instituted  a  new  form  of  organization  to  implement 
ISD,  but  rather  has  concentrated  on  teaching  ISD  procedures  in  its  instructor  training 
courses.  Formal  training  is  developed  at  the  resident  school  where  it  is  conducted, 
primarily  by  course  instructors.  (Civilian  education  specialists  also  participate  in 
course  design  to  some  degree,  especially  in  the  preparation  of  course  control  docu¬ 
ments  for  approval.)  The  Marine  Corps  policy  is  to  require  all  instructors  to  apply 
ISD  whenever  they  are  developing  a  new  course,  but  not  to  revise  existing  training 
that  had  been  developed  in  accordance  with  the  Marine  Corps’  earlier  nine-step  “sys¬ 
tems  approach”  model  for  training  development. 

The  extent  of  application  of  ISD  has  been  quite  limited.  There  are  instances, 
however,  in  which  training  has  been  redesigned  or  developed  using  some  of  the  ISD 
steps.  The  personnel  involved  ranged  from  one  instructor  or  education  specialist  to  an 
ad  hoc  team  of  several  instructors,  sometimes  organized  under  the  name  of  a  Course 
Content  Review  Board.  When  such  teams  are  used  to  design  training,  the  actual 
authoring  of  instruction  (generally  lesson  plans)  is  still  generally  accomplished  by  the 
instructors  who  will  use  it. 


AIR  FORCE 

Technical  training  by  instructors  in  the  Air  Force  is  developed  at  the  resident 
schools  where  it  is  conducted.  Instructor  training  has  therefore  been  revised  to 
include  ISD  techniques.  In  addition,  an  ISD  Specialty  has  been  added  to  the  Air 
Force  job  structure.  ISD  Specialists  and  civilian  education  specialists  provide  guidance 
to  the  instructors  who  are  developing  or  revising  their  courses,  and  they  coordinate 
approval  of  the  course  objectives  (Program  of  Instruction)  by  Air  Training  Command 
(ATC).  At  the  time  of  this  report,  the  future  of  the  ISD  specialty  is  in  some  doubt. 

The  application  of  ISD  to  Air  Force  technical  training  is  greatly  conditioned  by 
certain  components  of  the  existing  training  development  system.  An  occupational 
measurement  squadron  is  responsible  for  providing  job  task  lists  to  schools.  For  each 


'Military  Specification  MIL-T-29053,  Training  Requirements  for  Aviation  Weapon  Systems.  1977. 
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course,  schools  are  required  to  publish  a  Course  or  Specialty  Training  Standard  that 
lists  tasks  selected  for  training.  A  list  of  training  objectives  must  be  published  in  the 
form  of  a  Program  of  Instruction.  To  a  large  extent,  the  application  of  ISD  in  tech¬ 
nical  training  consists  of  satisfying  these  documentation  requirements. 

The  application  of  ISD  to  aircrew  training  is  more  centralized.  For  each  aircraft, 
training  is  developed  at  a  single  site,  although  it  may  be  conducted  at  several.  As  in 
Navy  flying  training,  major  ISD  efforts  are  being  undertaken  to  develop  individualized 
(usually  audiovisual)  instructional  materials  and  to  substitute  simulator  hours  for  flight 
time.  The  typical  approach  is  the  use  of  an  ISD  team,  either  from  an  ISD  squadron 
or  as  a  part  of  the  flying  training  squadron  or  wing  where  the  training  is  being  devel¬ 
oped.  Although  team  members  may  be  aircrew  personnel  assigned  to  ISD  duties,  they 
are  not  the  flight  line  instructors  who  will  teach  the  course.  This  degree  of  separation 
of  functions  has  the  potential  to  increase  the  degree  of  objectivity  in  the  job  and  train¬ 
ing  analysis,  but  also  presents  the  problems  of  communication  and  resistance  inherent 
in  any  effort  to  separate  training  development  from  implementation.1 
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SUMMATION 

No  particular  method  of  organizing  for  ISD  (centralization  or  decentralization, 
separation  of  functions  or  unitary  approach)  was  found  to  promote  or  hinder  its 
application.  To  a  certain  extent  this  finding  reflects  several  difficulties  encountered 
in  trying  to  assess  the  relative  effectiveness  or  efficiency  of  alternate  approaches: 

(1)  None  of  the  training  design  efforts  examined  followed  ISD  procedures 
closely  enough  so  that  the  adequacy  of  the  resulting  training  could  be 
considered  a  measure  of  the  adequacy  of  ISD,  much  less  of  a  particu¬ 
lar  approach. 

(2)  Data  on  the  cost  of  applying  ISD,  on  which  to  base  comparisons  of 
efficiency,  were  seldom  available. 

(3)  The  ISD  efforts  that  were  examined  cover  a  range  of  types  of  training 
in  different  content  areas.  Measuring  the  effectiveness  of  alternate 
organizational  approaches  would  require  that  the  costs  and  effects 
associated  with  the  way  the  ISD  process  has  been  structured  be  identi¬ 
fied  and  isolated  from  the  costs  and  effects  associated  with  particular 
training  methods  or  jobs. 

These  difficulties  notwithstanding,  the  research  team’s  conclusion  after  examining 
57  courses  is  that  whether  or  not  ISD  procedures  are  closely  followed  is  not  a  func¬ 
tion  of  the  manner  in  which  people  are  organized  to  apply  it.  At  least  within  the 
range  of  approaches  represented  in  the  study,  none  was  found  to  be  superior. 

It  was  also  found  that  separating  ISD  functions  does  not  of  itself  insure  a  high 
degree  of  objectivity.  When  the  effort  is  made  to  insulate  one  ISD  activity  from  the 
influence  of  another  or  from  past  practice,  by  handing  off  ISD  products  from  one 
party  to  another,  the  products  are  often  modified  to  suit  those  who  receive  them. 
This  applies  to  hands-offs  both  among  instructional  developers  and  from  developers 
to  instructors. 

i 


'See  Miller,  Ralph  M.,  Swink,  Jay  R.,  and  McKenzie,  James  F.,  Jr.,  Instructional  Systems 
Development  (ISD)  in  Air  Force  Flying  Training,  Air  Force  Human  Resources  Laboratory 
TR-78-59,  1978. 
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ISD  is  typically  performed  by  a  small  team.  Even  when  some  other  form  of  organi¬ 
zation  has  been  formally  established  to  carry  out  ISD,  there  is  a  tendency  to  revert  to 
a  small  team.  In  the  Marine  Corps  and  in  flying  training  the  approach  is  standard; 
in  other  instances  teams  are  often  assembled  ad  hoc. 

ISD  is  generally  performed  by  instructors.  Of  the  57  training  design  efforts  studied, 
instructors  were  primarily  responsible  for  39.  Moreover,  even  when  analysis  and  design 
are  accomplished  by  others,  the  actual  authoring  of  instruction  is  often  performed  by 
those  who  teach  the  course. 
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Chapter  5 

FIELD  STUDY  OF  ISD  APPLICATIONS 


FIELD  STUDY  OF  COURSES 

Information  about  how  ISD  is  being  applied  was  obtained  by  examining  the  proce¬ 
dures  used  in  developing  or  revising  57  courses  at  33  organizations  in  the  four  Services. 
Training  developers  who  had  participated  in  course  design  described  how  training  had  been 
developed  and  implemented  in  each  case.  The  initial  identification  and  screening  of  instal¬ 
lations  to  be  visited  and  courses  to  be  examined  was  made  on  the  basis  of  information 
obtained  in  the  questionnaire  survey  described  in  Chapter  3.  The  strategy  used  in  identifying 
sites  was  to  seek  out  courses  that  provided  the  most  complete  representation  of  the  ISD 
process.  This  appraisal  was  verified  in  phone  conversations  prior  to  a  visit  and  again  upon 
arrival  at  the  site.  Occasionally  a  course  that  had  been  previously  selected  was  deleted, 
and  one  that  developers  felt  represented  a  better  example  of  their  best  efforts  to  employ 
ISD  was  substituted.  The  sample  of  courses  that  was  examined  represents  the  best  exam¬ 
ples  of  ISD  that  were  identified.  Thus,  the  sample  is  exemplary  rather  than  typical. 

In  many  cases  several  persons  provided  information  about  a  given  course.  Each  step  of 
the  design  process  that  had  occurred  was  first  identified;  then  the  procedures  and  products 
for  each  step  were  described  and  discussed  in  detail.  Each  interview  covered  only  a  single 
course  and  how  it  was  developed.  This  approach  resulted  in  a  body  of  data  consisting  of  a 
collection  of  specifics  rather  than  a  collection  of  general  procedures.  A  structured  format  to 
the  interview  provided  for  cross-questioning,  to  check  on  the  accuracy  of  the  information.1 
In  almost  all  instances,  however,  interviewees  proved  to  be  quite  candid  in  their  descrip¬ 
tions  of  what  they  had  and  had  not  done. 

As  indicated  in  Table  10,  courses  were  examined  in  each  Service.  Combat,  air  crew,  and 
technical  training,  both  maintenance  and  no  -maintenance,  are  represented.  The  sampling 
of  courses,  however,  is  not  related  to  such  factors  as  combat  importance  or  input  density. 
Instead,  an  effort  was  made  to  include  examples  of  as  wide  a  range  of  types  of  training  as 
practical.  Intentionally  avoided  were  courses  with  a  strong  educational  and  informational 
orientation,  non-jol  -specific  courses,  and  others  in  which  it  is  difficult  to  identify  specific 
behavioral  objectives  (e.g.,  Service  academies,  leadership,  organizational  effectiveness). 

The  remainder  of  this  chapter  contains  a  step-by-step  description  of  the  way  in  which 
the  ISD  process  is  being  applied.  It  is  organized  like  Chapter  2,  discussing  each  ISD  step  in 
turn.  Following  a  summary  description  of  each  step5  are  a  table  listing  prerequisites  and 
procedures  for  carrying  out  the  step;  statistics  on  the  number  of  courses  in  which  the  pre¬ 
requisites  were  present  and  the  procedures  were  carried  out;  and  a  general  description  of 
the  way  in  which  the  step  has  been  applied.  The  chapter  concludes  with  a  general  discussion 
of  how  ISD  is  being  used.  In  Appendix  B,  the  data  on  the  individual  steps  summarized  for 
all  Services  in  this  chapter  are  broken  down  by  the  individual  Services. 


1  ( 'opM  of  the  interview  guide  can  be  obtained  from  HumRRO,  Western  Division. 

Earl  step,  previously  described  in  Chapter  2,  has  been  summarized  so  that  the  present  chapter 
>  ml  •pendent  I  v  TSe  reader  who  has  read  Chapter  2  may  wish  to  skip  over  the  introductory 


Table  10 


Courses  Examined 


ARMY 

•  Short  Range  Air  Defense  Missile  Crewman 

-  Air  Defense  School,  Fort  Bliss,  Texas 

•  Infantry  Officer  Advanced 

-  Infantry  School,  Fort  Benning,  Georgia 

•  Field  Artillery  Crewman 

-  Field  Artillery  School,  Fort  Sill,  Oklahoma 

•  Aerial  Surveillance  Sensor  Repairer 

-  Intelligence  School,  Fort  Huachuca,  Arizona 

•  Tracked  Vehicle  Mechanic 

-  Armor  School,  Fort  Knox,  Kentucky 

•  Improved  Hawk  Fire  Control  Repairman 

•  Nuclear  Weapons  Electronic  Specialist 

-  Missile  and  Munitions  School,  Redstone  Arsenal,  Alabama 

NAVY 

•  Avionics  Technician 

•  Basic  Electricity  and  Electronics 

-  Naval  Air  Technical  Training  Center,  Memphis,  Tennessee 

•  Aviation  Storekeeper 

•  Aviation  Maintenance  Administrationman 

•  Personnelman 

-  Naval  Technical  Training  Center,  Meridian,  Mississippi 

•  Propulsion  Engineering 

•  Fire  Control  Technician 

-  Service  Schools  Command,  Great  Lakes,  Illinois 

•  Radio  man 

•  Internal  Communicationman 

-  Instructional  Program  Development  Center,  San  Diego,  California 


Table  10  f Continued ) 
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•  F-4J  Electrical  Systems  Organizational  Maintenance 

—  Naval  Air  Maintenance  Training  Detachment  1024,  MCAS  Beaufort,  South,  Carolina 

•  AN/APM  225  Module  Analyzer  Test  Console 

—  Naval  Air  Maintenance  Training  Detachment  1003,  NAS  Oceana,  Virginia 

•  SH2F  Airframe  Maintenance 

—  Naval  Air  Maintenance  Training  Detachment  1007,  NAS  Norfolk,  Virginia 

•  Hagan  Automatic  Boiler  Control  Console  Operator 

•  Storekeeper  Supply  Afloat  Independent  Duty 

•  AN/SPS  49  Heavy  Radar  Maintenance 

—  Fleet  Training  Center,  Norfolk,  Virginia 

•  Nuclear  Weapons  Advanced  Maintenance 

—  Nuclear  Weapons  Training  Group  Atlantic,  Norfolk,  Virginia 

•  Surface  Anti-Submarine  Warfare  Officer 

—  Fleet  Anti-Submarine  Warfare  Training  Center  Atlantic,  Norfolk,  Virginia 

•  OK-252/BQR-15  Array  &  Cable  Handling  Group 

•  MK24  Hydraulic/Support  Ring  Advanced 

—  Fleet  Ballistic  Missile  Submarine  Training  Center,  Charleston,  South  Carolina 

•  P-3C/P-38  MOD  Aircrew 

—  Patrol  Squadron  31,  NAS  Moffet  Field,  California 

•  F14  Aircrew 

—  Fighter  Squadron  124,  NAS  Miramar,  California 

•  SH-3H  Aircrew 

—  Helicopter  Anti-Submarine  Squadron  10,  NAS  North  Island,  California 

•  Basic  Underwater  Demolition  and  SEAL 

—  Naval  Amphibious  School,  Coronado,  California 

MARINE  CORPS 

•  Instructional  Management 

•  IBM  360  Systems 

•  Communications  Officer 

—  Education  Center,  Development  and  Education  Command,  Quantico,  Virginia 

- - - — -  (Continued)  - 
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Table  10  (Continued) 


•  Motor  Transport  Staff  NCO 

—  Motor  Transport  School,  Camp  Lejeune,  North  Carolina 

•  Food  Service  Basic 

•  Fiscal  Accounting 

—  Service  Support  Schools,  Camp  Lejeune,  North  Carolina 

•  Journeyman  Combat  Engineer 

—  Engineer  School,  Camp  Lejeune,  North  Carolina 

AIR  FORCE 

•  Television  Equipment  Repairman 

•  Technical  Instructor 

•  Services  Operations  Officer 

•  Instrumentation  Mechanic:  Sets  Training 

•  Material  Facilities  Specialist 

—  Lowry  Technical  Training  Center,  Colorado 

•  Dental  Supervisor 

•  Optometry  Specialist 

•  Mental  Health  Ward  Specialist 

—  School  of  Health  Care  Sciences,  Sheppard  AFB,  Texas 

•  Aircraft  Propeller  Mechanic 

•  Contract  Construction  Inspector 

•  Reciprocating  Aircraft  Maintenance  Specialist 

•  Outside  Wire  and  Antenna  Repairman 

•  Accounting  and  Finance  Officer 

—  Sheppard  Technical  Training  Center,  Texas 

•  E-3A  Avionics  Instrument  Specialist 

•  E-3A  Electrical  System  Specialist 

—  Field  Training  Detachment  413,  Tinker  AFB,  Oklahoma 

•  E-3A  Weapons  Director 

•  E-3A  Airborne  Computer/Display  Maintenance  Technician 

—  552  AWAC  Wing,  Tinker  AFB,  Oklahoma 

•  AQM-34M  Missile  Systems  Maintenance  Specialist 

—  Field  Training  Detachment  512,  Davis-Monthan  AFB,  Arizona 

•  A-10  Pilot 

—  4444  Operations  Squadron  (TAC),  Davis-Monthan  AFB,  Arizona 

•  H-53  Combat  Rescue  Aircrew 

—  1550  Aircrew  Training  and  Test  Wing  (MAC),  Kirtland  AFB,  New  Mexico 
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STEPS  OF  PROCESS 


Need/Discrepancy  Analysis 

ISD  originates  with  the  identification  of  a  need  to  develop  or  revise  instruction. 
Instruction  must  be  developed  if  the  human  performance  elements  of  a  new  job  or 
weapon  system  require  training.  Instruction  must  be  revised  if  existing  training  is  not 
sufficiently  effective  and  economical.  Prerequisite  conditions  for  determining  need  are 
the  existence  and  maintenance  of  procedures  for  detecting  indications  of  potential  dis¬ 
crepancies.  When  a  potential  discrepancy  is  identified,  additional  information  is  gathered 
as  needed  for  further  confirmation  and  definition  of  the  problem,  and  to  suggest  the 
appropriate  locus  for  corrective  action— that  is,  where  the  ISD  process  should  begin,  and 
the  extent  of  ISD  that  is  appropriate  to  the  problem. 

Table  1 1 


Need/Discrepancy  Analysis: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N=57) 


Prerequisites 

Present 

Absent 

Not  Applicable3/ 

Not  Determined 

1 .  Information  is  available  about  possible  dis¬ 
crepancies  between  training  and  field 
requirements. 

31 

2 

24 

Procedures 

Done 

Not  Done 

Not  Applicable*)/ 

Not  Determined 

1 .  Analyze  initial  and  supplementary 
information. 

24 

22 

11 

2.  Identify  and  specify  discrepancy. 

23 

23 

11 

3.  Specify  ISD  entry  point  and  boundaries 
of  redesign  process. 

19 

27 

11 

aNot  Applicable  prerequisites  occur  in  instances  where  their  status  is  moot  (e.g„  prerequisite  states  the  location 
in  ISO  process  where  step  is  to  occur,  but  step  has  not  been  performed). 

bNot  Applicable  procedures  occur  in  ISD  efforts  that  were  examined  in  progress  and  for  which  a  given  step  was 
not  yet  appropriate. 


In  about  one-third  of  the  courses  studied,  it  was  found  that  training  development 
had  been  initiated  in  response  to  an  identified  training  deficiency,  and  that  the  entry 
point  for  ISD  had  been  specified  accordingly.  In  most  cases,  however,  the  deficiency 
was  simply  the  lack  of  any  training  program  for  some  newly  introduced  equipment  or 
a  new  job  specialty.  Only  five  of  the  development  efforts  (9%)  had  been  undertaken  in 
response  to  indications  of  poor  training  or  job  performance,  such  as  failed  inspections 
or  high  attrition.  In  11  courses,  it  could  not  be  determined  whether  the  identification 
of  a  training  deficiency  had  led  to  the  requirement  to  redesign  existing  training.  Train¬ 
ing  developers  interviewed  did  not  always  know  what  circumstances  preceded  their 
own  efforts. 
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The  low  incidence  of  cases  in  which  specific  training  deficiencies  were  identified 
and  remedied  through  the  ISD  process  may  be  partly  due  to  the  manner  of  selecting 
the  sample  of  courses  for  this  study.  Courses  which  had  reportedly  undergone  many 
ISD  steps  were  selected  over  those  which  had  undergone  only  a  few,  and  training  devel¬ 
opers  on  site  were  asked  to  describe  their  “most  complete”  ISD  efforts.  Both  of  these 
factors  would  tend  to  favor  the  selection  of  training  design  efforts  which  had  begun 
with  front-end  analysis,  rather  than  those  for  which  a  later  ISD  entry  point  had  been 
specified. 


Identification  of  Job  Requirements 


Job  analysis  consists  of  three  parts:  compiling  a  provisional  list  of  tasks  believed 
to  comprise  the  job;  verifying  the  accuracy  of  the  list  and  adjusting  it  as  necessary;  and 
gathering  task  priority  information.  Task  lists  are  developed  in  a  two-phase  process  to 
increase  their  reliability.  The  initial  list  may  be  generated  in  various  ways,  ranging  from 
the  recollections  of  a  single  subject  matter  expert  to  extensive  observation  of  job  incum¬ 
bents.  Verification  may  also  take  different  forms,  ranging  from  review  by  a  second  group 
of  subject  matter  experts  to  a  phone  or  much  broader  mail  survey.  Ideally,  verification 
includes  validation  with  a  representative  sample  of  job  incumbents. 

Task  priority  information  is  data  about  such  factors  as  billets  where  the  task  is  per¬ 
formed,  percent  of  persons  performing,  and  frequency  of  performance.  To  increase  the 
reliability  of  what  are  inevitably  subjective  estimates,  this  information  should  usually  be 
gathered  from  a  fairly  large  sample  of  job  incumbents. 

Table  12 


Identification  of  Job  Requirements: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N=57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 

Not  Determined 

None 

Procedures 

Done 

Not  Done 

Not  Applicable 

Not  Determined 

1.  Construct  provisional  task  list. 

38 

13 

6 

2.  Verify  and  revise  on  basis  of  review/ 
survey  of  job  incumbents. 

27 

20 

10 

3.  Collect  task  priority  information. 

24 

23 

10 

A  provisional  task  list  was  constructed  in  38  of  the  courses  studied,  and  in  27  of 
those  cases  it  was  subjected  to  some  kind  of  verification  by  job  incumbents  or  other 
subject  matter  experts.  In  the  Army  most  of  the  lists  were  constructed  by  personnel 
permanently  assigned  to  task  analysis  duties  at  TRADOC  schools,  as  in  a  Directorate  of 
Training  Developments.  In  the  Marine  Corps  the  lists  were  generally  constructed  by 
resident  school  instructors  temporarily  assigned  to  course  development  duties,  sometimes 
as  members  of  an  ad  hoc  Course  Content  Review  Board.  In  the  Navy  the  list  was  likely 
to  be  an  occupational  survey  (NOTAP)  printout,  or  an  existing  list  of  watch-standing 
or  equipment  maintenance  duties.  When  a  Navy  task  list  was  constructed  specifically 
for  course  development,  this  was  usually  done,  in  the  case  of  technical  training,  by 
instructors  under  the  coordination  of  a  Curriculum  and  Instructional  Standards  Office, 
or,  for  air  crew  training,  by  former  incumbents  temporarily  assigned  to  work  with  an 
ISD  contractor.  Most  Air  Force  task  lists  were  CODAP  printouts.  When  a  list  was  con¬ 
structed  by  Air  Force  course  developers,  which  occurred  only  rarely  in  non-flying  train¬ 
ing,  it  was  compiled  by  instructors. 

The  principal  instances  in  which  job  task  lists  were  not  compiled  were  a  few  field- 
conducted,  equipment-specific  courses  in  the  Navy  and  a  variety  of  Air  Force  courses 
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for  which  no  occupational  survey  report  was  available.  Of  the  development  efforts 
examined,  those  in  the  Air  Force  had  the  highest  (50%)  of  cases  in  which  task  lists 
were  definitely  not  compiled.  It  may  be  that  the  wide  availability  of  CODAP  printouts 
leads  Air  Force  training  designers  to  consider  task  list  construction  solely  the  respon¬ 
sibility  of  occupational  measurement  personnel.  The  procedure  specified  in  Air  Force 
Pamphlet  50-58,  however,  is  to  use  occupational  survey  data  along  with  other  information 
to  build  a  complete  task  list  arranged  by  job  duties.  This  approach  was  rarely  taken 
in  Air  Force  technical  training.  If  a  CODAP  printout  had  been  obtained,  the  develop¬ 
ment  of  a  Course  or  Specialty  Training  Standard  (selection  of  tasks  for  training)  was 
generally  reported  as  the  next  ISD  step  accomplished. 

If  either  a  NOTAP  or  a  CODAP  printout  was  available  and  reported  to  be  used,  it 
was  considered  a  verified  task  list  for  the  purposes  of  this  study.  However,  personnel 
interviewed  often  questioned  the  usefulness  of  these  occupational  survey  reports,  on 
the  grounds  that  the  task  descriptions  were  not  suitable  for  deriving  training  objectives. 

As  the  recommended  AFP  50-58  procedure  implies,  perhaps  such  lists  are  not  adequate 
by  themselves  to  serve  as  the  basis  for  training  development.  Examination  of  some  of 
the  NOTAP/CODAP  lists  used  to  develop  courses  in  this  study  revealed  that  some  items 
listed  were  well-defined  task  descriptions,  while  others  were  actually  general  skills  or  non¬ 
task  categories  of  behavior. 


Selection  of  Tasks  for  Training 


Tasks  identified  by  job  analysis  are  classified  into  those  to  be  trained  and  those  not 
to  be  trained.  Tasks  selected  form  the  basis  of  the  training  program;  those  rejected  are 
not  reconsidered  until  the  final  external  evaluation.  Prerequisites  for  task  selection  are 
a  list  of  tasks  which  comprise  the  job,  information  for  establishing  their  importance  and 
need  for  training,  and  decision  rules  for  determining  training  priorities.  To  promote  objec¬ 
tivity,  this  information  and  the  procedures  for  processing  it  should  be  developed  prior  to 
the  selection  process.  After  identifying  which  tasks  are  desirable  to  train,  and  which  tasks 
are  most  important,  requirements  for  training  resources  are  considered.  Final  selection 
is  made  on  the  basis  of  training  resource  requirements  and  availability— that  is,  cost.  A 
factor  which  greatly  influences  cost  is  the  training  setting,  but  to  avoid  bias  in  task  selec¬ 
tion  the  setting  is  selected  separately,  insofar  as  possible. 

Table  13 

Selection  of  Tasks  for  Training: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N-57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 
Not  Determined 

1.  Tasks  performed  in  the  job  are  listed. 

38 

13 

6 

2.  Information  is  available  for  establishing 

importance  of  tasks  and  need  for  training. 

23 

23 

11 

3.  Decision  rules  to  be  applied  to  task 
information  are  available. 

7 

40 

10 

Procedures 

Done 

Not  Done 

Not  Applicable/ 
N^t  Determined 

1 .  Apply  decision  rules  to  information  for  each 

task  to  determine  training  priorities. 

5 

41 

11 

2.  Select  tasks  for  training  on  basis  of  training 
priorities  and  resource  availability. 

6 

40 

11 

Although  a  job  task  list  of  some  kind  was  available  in  38  of  the  courses  studied, 
information  for  establishing  task  importance  was  available  in  only  23,  and  decision 
rules  for  selecting  tasks  for  training  were  applied  to  this  information  in  only  five.  More¬ 
over,  such  decision  rules  as  were  followed  were  rudimentary  at  best.  In  one  case,  the 
rule  of  thumb  was  to  train  any  task  which  at  least  5%  of  incumbents  performed;  in 
another  it  was  30%;  in  a  third,  it  was  tasks  judged  critical  to  mission  accomplishment. 

In  a  fourth  instance,  tasks  were  generally  chosen  if  performed  by  60%  or  more  of 
incumbents,  or  if  critical  or  difficult. 

Overall,  the  selection  of  tasks  for  training  in  the  four  Services  was  not  characterized 
by  deliberate  determination  of  task  selection  criteria  and  the  subsequent  selection  of 
tasks  on  the  basis  of  task  priority  information.  When  priority  information  was  collected, 
it  was  rarely  employed  in  any  systematic  fashion.  When  training  designers  reported  that 
such  information  was  used  in  selecting  tasks,  it  was  common  to  find  that  it  had  been 
furnished  by  the  same  subject  matter  experts  who  were  making  the  task  selection,  and 
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at  the  same  time.  Even  when  the  effort  was  made  to  collect  task  priority  information 
from  job  incumbents  independently,  respondents  were  often  asked  to  indicate  the  cri¬ 
ticality,  difficulty,  frequency,  and  so  forth  of  a  task  and  at  the  same  time  whether  it 
should  be  trained— thus  confounding  job  analysis  with  task  selection,  and  losing  the 
advantages  in  objectivity  of  keeping  the  activities  separate.  Under  these  conditions,  one 
task  might  be  selected  for  its  frequency,  and  another  for  its  difficulty.  Applying  the 
same  criteria  to  all  tasks,  or  certain  criteria  to  particular  classes  of  tasks,  rarely,  if  ever, 
occurred.  Tasks  were  seldom  ranked  in  importance  to  allow  the  most  efficient  alloca¬ 
tion  of  available  training  resources  or  to  aid  in  determining  what  those  resources  should 
be.  Thus,  any  potential  of  the  ISD  approach  for  determining  training  resource  require¬ 
ments  according  to  task  importance  was  not  generally  being  realized. 

Aside  from  the  absence  of  decision  rules,  other  practices  tended  to  make  task  selec¬ 
tion  less  than  systematic.  Often,  the  original  job  task  list  was  not  maintained  in  readily 
usable  form.  Once  tasks  were  selected,  it  was  difficult  to  determine  where  they  had 
come  from.  In  other  cases,  task  selection  began  before  job  analysis  was  complete,  or  was 
substituted  for  it  altogether.  This  may  have  been  due  to  a  natural  reluctance  to  expend 
effort  delineating  tasks  felt  to  be  unimportant,  or  to  management  emphasis  on  records 
of  tasks  selected  for  training  (e.g.,  Training  Standards,  Soldier’s  Manuals)  rather  than  job 
task  inventories. 

Records  showing  which  job  tasks  had  been  selected  and  which  rejected  were  seldom 
available,  and  records  showing  why  a  given  task  had  been  selected,  hardly  ever.  Tasks 
were  commonly  combined,  subdivided,  or  otherwise  altered  in  the  process  of  going  from 
job  requirements  to  training  requirements,  especially  when  the  task  list  was  an  occupa¬ 
tional  survey  report.  Often  the  only  evidence  of  which  tasks  had  been  selected  was  a  set 
of  training  objectives,  each  of  which  might  cover  several  tasks,  or  only  part  of  one.  In 
the  absence  of  records  to  the  contrary,  it  is  possible  that  the  tasks  “selected”  for  training 
in  a  given  course  were  the  only  tasks  considered— that  is,  selection  would  not  have 
occurred. 

In  the  Army,  tasks  were  selected  by  personnel  assigned  to  job  analysis  duties  in  the 
Directorate  of  Training  Developments  at  a  TRADOC  school,  who  were  responsible  for 
developing  Soldier’s  Manuals.  After  tasks  had  been  selected  for  this  purpose,  the  same 
task  lists  were  forwarded  to  other  directorates  or  divisions  within  the  school  to  serve  as 
the  basis  for  TEC  lessons.  Skill  Qualification  Tests,  resident  training,  and  other  training 
elements.  It  was  found  that  the  selection  of  tasks  made  by  those  responsible  for  job 
analysis  was  not  always  accepted  and  used  by  those  to  whom  it  was  forwarded.  This 
was  especially  likely  to  happen  in  the  development  of  resident  courses,  where  in  some 
instances  the  activities  conducted  at  the  “front  end”  of  the  ISD  process  had  little  or  no 
connection  with  what  was  taught  in  the  classroom.  This  situation  was  not  universal 
across  the  six  Army  schools  visited,  but  it  was  typical. 

It  should  be  noted  that  the  Army  is  currently  emphasizing  training  and  evaluation 
products  for  use  in  operational  units,  rather  than  in  schools  (Soldier’s  Manuals,  TEC 
lessons,  SQTs,  etc.).  With  respect  to  resident  training,  more  emphasis  is  being  given  to 
changing  training  methods  than  to  determining  content.  ISD  activities  were  being  applied 
to  resident  training  primarily  when  courses  were  being  converted  to  self-pacing. 

Since  task  selection  in  the  Army  is  performed  at  the  school  which  is  proponent  for 
doctrine  in  a  particular  MOS,  command  approval  is  inherent  in  the  selection  process.  In 
the  Navy,  Marine  Corps,  and  Air  Force,  tasks  selected  for  training  (usually  in  the  form 
of  training  objectives)  are  typically  sent  to  a  higher  command  for  approval.  In  the  Navy 
and  Air  Force,  where  training  developed  at  one  site  may  be  conducted  at  others  as  well, 
and  graduates  of  a  course  may  be  assigned  to  different  commands,  it  is  common  to 
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seek  the  concurrence  of  the  other  units  or  commands  involved.  In  the  Air  Force,  a 
training  review  conference  (“scrubdown”)  is  sometimes  convened,  bringing  representatives 
of  receiving  commands  together  with  representatives  of  the  school  where  initial  training 
is  developed  and  implemented. 

Efforts  to  obtain  agreement  on  training  content  represent  an  apparent  departure 
from  ISD  methodology,  which  suggests  the  use  of  rigorous  decision  rules,  rather  than 
the  achievement  of  consensus.  Personal  judgment  may,  however,  be  the  only  means  for 
task  selection  now  available.  As  noted  in  Chapter  2,  there  are  currently  no  known  cri¬ 
teria  of  total  system  effectiveness  by  which  to  determine  task  selection  criteria.  Perhaps 
the  almost  universal  failure  of  training  designers  to  select  tasks  systematically  according 
to  predetermined  criteria  is  in  recognition  of  how  arbitrary  any  set  of  task  selection  rules 
must  be.  Even  when  given  a  mathematical  formula  to  apply  to  available  information,  as  in 
Air  Force  Pamphlet  50-58,  no  one  reported  using  it. 

In  the  absence  of  decision  rules  for  task  selection,  there  may  nevertheless  be  some 
value  in  requiring  task  priority  information  to  be  collected  by  survey,  or  even  “developed” 
when  tasks  are  selected.  The  mere  availability  of  such  information  may  make  the  per¬ 
sonal  judgment  of  training  designers  more  informed,  and  the  requirement  to  justify  task 
selection  on  some  grounds  may  make  the  selection  more  thoughtful. 


Analysis  of  Tasks 


Task  analysis  is  a  description  of  when  and  how,  within  the  job  environment,  perform¬ 
ance  of  a  task  is  required.  It  consists  of  \  specification  of  conditions  of  performance  and 
initiating  cues,  behavioral  elements,  and  standards  of  performance.  Information  generated 
during  task  analysis  is  used  later  in  constructing  Job  Performance  Measures  and  developing 
objectives.  The  prerequisite  for  task  analysis  is  the  identification  of  tasks  to  be  analyzed. 

Table  14 

Analysis  of  Tasks: 

Incidence  of  Prerequisites  and  Procedures  in  Course  Examined  (N=57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 
Not  Determined 

1.  Tasks  selected  for  training  are  listed. 

41 

10 

6 

Procedures 

Done 

Not  Done 

Not  Applicable/ 
Not  Determined 

1 .  For  each  task  specify  for  the  job 

environment: 

—  Conditions  of  performance. 

16 

29 

12 

—  Behavioral  elements. 

31 

13 

13 

—  Standards  of  performance. 

19 

28 

10 

In  the  41  training  design  efforts  in  which  tasks  were  identified  for  training,  behavioral 
elements  were  identified  in  about  three-fourths  (31  courses),  while  task  conditions  and 
standards  were  identified  in  less  than  half.  The  greater  incidence  of  identification  of 
behavioral  elements  was  partly  due  to  their  being  listed  in  training  objectives  or  instruc¬ 
tional  materials.  The  Army  and  Marines  account,  proportionally,  for  nearly  all  explicit 
task  analysis.  In  the  Army,  task  conditions,  elements,  and  standards  are  listed  in  the 
Soldier’s  Manual;  in  the  Marine  Corps  they  were  found  in  task  analysis  worksheets.  In 
the  Navy  and  Air  Force,  the  identification  of  task  conditions  and  standards  as  distinct 
from  training  conditions  and  standards  was  infrequent.  The  approach  generally  followed 
was  to  specify  conditions  only  as  required  for  course  control  documents,  without  first 
having  determined  the  standards  and  conditions  of  job  performance.  Two  of  the  four 
instances  of  explicit  task  analysis  found  in  the  Navy  were  large-scale  ISD  efforts  at  the 
Instructional  Program  Development  Center,  San  Diego. 
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Construction  of  Job  Performance  Measures 

Job  Performance  Measures  may  be  constructed  for  each  task  that  has  been  selected 
for  training  to  serve  as  a  means  both  of  keeping  training  faithful  to  job  requirements  and 
of  evaluating  training  and  the  design  process.  To  maintain  fidelity  to  job  requirements, 
training  objectives  can  be  derived  directly  from  Job  Performance  Measures.  To  evaluate 
training  design,  Job  Performance  Measures  are  administered  to  persons  in  training,  or  after 
they  have  reached  the  job.  To  serve  their  purposes,  Job  Performance  Measures  must  be 
constructed  before  objectives  or  achievement  tests  are  developed.  Not  all  models  of  ISD 
include  the  construction  of  Job  Performance  Measures. 

Table  15 

Construction  of  Job  Performance  Measures: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N=57) 


Prerequisites 

1.  Tasks  selected  for  training  are  listed. 

2.  Training  requirements  for  these  tasks 
have  not  been  identified. 

Procedures 

1 .  Construct  a  test  for  measuring  the  per¬ 
formance  of  each  task  selected  for  training. 

2.  Validate  each  Job  Performance  Measure  to 
insure  that  it  predicts  task  performance. 


Present 

Absent 

Not  Applicable/ 
Not  Determined 

12 

2 

43 

8 

6 

43 

Done 

Not  Done 

Not  Applicable/ 
Not  Determined 

71 

36 

10 

3 

38 

16 

Job  Performance  Measures  were  developed  in  11  courses,  and  validated  or  verified 
in  three.  In  all  cases  but  one,  however,  these  measures  were  simply  performance-oriented 
achievement  tests  constructed  for  within-course  use.  In  only  one  instance,  IPDC  San 
Diego,  were  JPMs  constructed  and  at  least  partially  validated  in  the  field,  in  accordance 
with  the  ITRO  model  procedures.  No  other  case  was  found  in  which  training  designers 
constructed  task  measures  early  in  the  training  design  process  to  serve  as  a  means  of 
evaluating  subsequent  course  design  and  implementation.  Training  designers  reported  in 
some  cases  that  they  did  not  have  sufficient  resources  to  attempt  to  construct  or  validate 
JPMs;  the  IPDC  itself  discontinued  the  attempt  after  its  first  course  development  effort, 
citing  the  considerable  time  and  expense  required.  The  remainder  of  the  courses  that 
were  classified  in  this  study  as  using  JPMs  generally  incorporated  exercises  performed  on 
actual  equipment  or  full-scale  training  devices— maintenance  simulators,  mock  warehouse, 
aircraft,  and  so  forth. 
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Selection  of  Setting 

Every  task  that  has  been  selected  for  training  must  be  assigned  to  a  particular  location 
and  situation  in  which  training  will  occur.  A  prerequisite  to  carrying  out  this  step  is  the 
freedom  to  assign  tasks  to  any  of  several  settings.  Another  is  the  availability  of  information 
about  the  costs  of  training  in  the  alternative  settings.  This  step  involves  an  interface  with 
other  components  of  the  operational  system,  beyond  the  training  subsystem.  To  assign 
tasks  to  different  settings  in  such  a  way  as  to  maximize  total  system  effectiveness  requires 
knowledge  of  how  alternative  assignment  patterns  would  affect  the  total  system. 

Table  16 

Selection  of  Setting: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N=57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 
Not  Determined 

1 .  Tasks  can  be  assigned  to  any  of 
several  settings. 

18 

23 

16 

2.  Information  is  available  on  costs  of  training 
in  different  settings. 

0 

0 

57 

3.  Information  is  available  on  effects  of 
training  in  different  settirgs  on  total  sys¬ 
tem  effectiveness. 

0 

0 

57 

Procedures 

Done 

Not  Done 

Not  Applicable/ 
Not  Determined 

1 .  Assign  each  task  or  group  of  tasks 
to  its  appropriate  setting. 

16 

37 

4 

In  only  18  courses  were  training  designers  free  to  assign  tasks  to  any  of  several  settings. 
Usually  they  were  concerned  only  with  the  setting  in  which  they  themselves  were  located 
and  for  which  they  were  developing  training.  In  the  Marine  Corps  no  instances  were  found 
in  which  training  designers  assigned  tasks  to  other  than  resident  schools,  and  in  both  the 
Navy  and  the  Air  Force  this  occurred  in  only  about  one-third  of  the  applicable  courses. 
Although  training  developers  in  the  Army  were  free  to  assign  tasks  to  different  settings  in 
three  out  of  four  applicable  cases,  in  one  of  these  the  procedure  was  simply  to  assign  every 
task  to  a  resident  school  unless  it  was  impractical  to  teach  it  there. 

In  almost  all  cases,  setting  selection  was  actually  synonymous  with  task  selection; 
personnel  selecting  tasks  for  training  were  almost  always  selecting  them  for  one  particular 
setting.  The  statement  in  the  AFP  50-58  model  that  course  developers  would  seldom  be 
in  a  position  to  designate  training  settings  (see  p.  32)  was  found  to  be  closer  to  actual 
practice  than  the  situation  depicted  by  the  ITRO  model,  in  which  designers  are  free  to 
develop  job  performance  aids  or  on-the-job  training  as  alternatives  to  school  training.  An 
exception  to  the  general  case  was  the  practice  in  some  Navy  training  design  efforts  of  desig¬ 
nating  “A”  School,  “C"  School,  formal  on-the-job  training,  or  other  setting  for  each  job  task. 
Since  personnel  interviewed  were  cognizant  only  of  how  training  had  been  developed  for 
their  own  setting,  it  was  not  determined  whether  training  was  actually  developed  for  these 
other  settings  as  well. 
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Development  of  Treining  Objectives  and 
Objectives  Hierarchies 

Training  objectives  are  descriptions  of  what  a  trainee  is  expected  to  be  able  to  do 
following  instruction.  Their  development  represents  a  shift  in  the  ISD  process  from  the 
job  to  training.  Developing  objectives  involves  serveral  steps: 

•  Deciding  how  closely  capabilities  for  task  performance  at  the  conclusion 
of  training  shall  match  the  requirements  of  the  job. 

•  Specifying  dependent  and  coordinate  relationships  within  tasks  (terminal 
objectives),  among  subtasks  (intermediate  objectives),  and  mediating  skills 
and  knowledge  (intermediate  objectives). 

•  Deciding,  based  on  estimates  of  the  abilities  of  entering  trainees,  what  ter¬ 
minal  objectives  must  actually  be  developed  during  training,  and  what 
knowledge  and  basic  skills  must  be  provided  to  mediate  learning  and 
performance. 

Table  17 


Development  of  Training  Objectives  and  Objectives  Hierarchies: 
Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N=57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 
Not  Determined 

1 .  Tasks  selected  for  training  are  listed. 

44 

7 

6 

2.  Information  is  available  about  training 
constraints  that  make  it  necessary  to 
modify  task  requirements. 

0 

0 

57 

3.  Information  is  available  about  how  modi¬ 
fication  of  task  requirements  will  affect 
training  efficiency. 

0 

0 

57 

4.  Estimates  of  capabilities  of  entering 
trainees  for  learning  and  performing  each 
task  are  available. 

5 

2 

50 

Procedures 

Done 

Not  Done 

Not  Applicable/ 
Not  Determined 

1.  Specify  task  requirements  (behaviors, 
conditions,  standards)  for  training. 

51 

1 

5 

2.  Perform  hierarchical  analysis  of  tasks  to 
identify  intermediate  training  objectives. 

14 

39 

4 

Training  objectives  were  developed  in  nearly  all  cases  (51  courses).  The  use  of 
behavioral  objectives  to  describe  training  content  is  thus  the  single  most  unifying  charac¬ 
teristic  of  the  courses  studied.  In  some  cases,  the  very  purpose  of  course  revision  was  to 
add  properly  worded  objectives  (e.g.,  observable,  measurable)  to  course  control  documents, 
The  relationship  of  objectives  to  job  tasks,  however,  varied  considerably.  In  several 
cases  objectives  were  developed  in  the  absence  of  any  specification  of  a  set  of  tasks 
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selected  for  training.  In  the  remaining  cases  it  was  common  for  one  objective  to  repre¬ 
sent  several  tasks,  or  vice  versa,  or  both.  In  Navy  and  Air  Force  technical  training  this 
was  standard  practice,  with  a  single  objective  sometimes  being  listed  as  pertaining  to  a 
dozen  or  more  different  tasks.  It  was  often  unclear  what  rules,  if  any,  had  governed  the 
derivation  of  objectives  from  a  single  task,  or  the  representation  of  so  many  tasks  by 
one  or  several  objectives.  In  many  cases  personnel  interviewed  did  not  report  the  use 
of  any  systematic  procedure.  Thus,  although  there  were  often  records  showing  which 
objectives  went  with  which  tasks,  there  is  reason  to  question  whether  the  objectives  had 
actually  been  derived  from  the  tasks. 

In  only  one-fourth  of  the  courses  studied  was  there  an  effort  to  develop  a  hierarchy 
to  derive  objectives.  In  the  great  majority  of  cases,  no  systematic  attempt  was  made  to 
analyze  tasks  or  other  objectives  to  determine  what  must  be  learned  in  order  to  master 
them.  Moreover,  in  the  cases  in  which  some  hierarchical  analysis  was  judged  to  have 
occurred,  an  actual  outline  or  diagram  of  the  relationship  of  objectives  to  tasks  or  other 
objectives  was  usually  absent.  Rarely  had  trainers  deliberately  estimated  the  capabilities 
of  entering  trainees  in  order  to  determine  the  limits  of  hierarchical  analysis. 

The  ISD  process  is  partly  a  means  of  insuring  that  training  will  not  exist  for  its  own 
sake,  but  that  mastery  of  training  requirements  will  lead  to  successful  job  performance. 

The  ISD  procedure  of  deriving  terminal  training  objectives  directly  from  tasks,  and  inter¬ 
mediate  or  enabling  objectives  in  turn,  is  intended  to  preserve  this  close  relationship.  In 
practice,  this  one-to-one  correspondence  between  job  tasks  and  terminal  objectives  was 
the  exception  rather  than  the  rule.  Although  there  is  nothing  inherently  unsystematic 
about  deriving  several  objectives  from  the  same  task,  or  combining  similar  tasks,  the 
typical  lack  of  any  corresponding  methodology  for  doing  so  suggests  that  objectives  were 
often  matched  to  tasks  after  training  content  was  already  known,  rather  than  as  the  means 
of  determining  that  content. 

It  appears  that  in  many  cases— perhaps  the  typical  case— objectives  had  been  written 
to  “cover”  subject  matter  currently  taught  or  being  added  to  a  course.  Under  this  con¬ 
dition,  training  objectives  would  have  been  from  training  content,  rather  than  content 
from  objectives.  Further  support  for  this  conclusion,  in  addition  to  the  fact  that  per¬ 
sonnel  interviewed  often  could  not  specify  the  derivation  process,  is  the  fact  that  objec¬ 
tives  often  were  written  simultaneously  with  instruction  and  by  the  same  persons.  In 
at  least  one  instance,  the  objectives  said  to  be  based  on  job  analysis  were  incorporated 
without  any  changes  in  what  was  taught. 

Job  conditions  were  specified  in  only  16  of  the  courses  studied,  and  job  standards 
in  19,  compared  to  the  51  courses  for  which  objectives  were  written.  Training  conditions 
and  standards,  then,  were  usually  specified  in  the  absence  of  explicit  job  conditions  and 
standards.  As  mentioned,  this  practice  was  found  more  frequently  in  the  Navy  and  Air 
Force.  Its  consequences  may  be  trivial,  if  training  designers  are  aware  of  job  standards 
and  conditions  and  develop  objectives  accordingly.  Just  as  easily,  however,  it  could  lead 
to  establishing  training  standards  that  are  unnecessarily  high,  or  so  low  that  training  is 
inadequate— the  very  conditions  that  the  systems  approach  is  designed  to  avoid. 

To  the  extent  that  objectives  are  developed  along  with  instruction,  as  they  often 
were,  the  strong  possibility  arises  that  training  standards  will  simply  be  written  to  reflect 
training  content.  As  the  data  show,  it  was  not  determined  how  much  training  designers 
knew  about  the  extent  of  changes  in  job  standards  or  conditions,  or  how  such  changes 
might  affect  training  effectiveness.  In  the  absence  of  specified  job  standards  or  conditions, 
of  course,  the  questions  are  moot.  When  task  analysis  had  been  accomplished,  some 
personnel  interviewed  were  aware  that  their  training  standards  differed  from  job  standards, 
but  no  one  reported  any  knowledge  of  the  possible  effects  of  the  change. 
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Development  of  Achievement  Tests 

Objective-referenced  achievement  tests  are  used  to  determine  the  capabilities  of  the 
entry  population,  to  determine  the  effectiveness  of  training,  and  to  diagnose  student  per¬ 
formance  during  training.  A  particular  requirement  of  ISD  is  that  tests  be  developed 
directly  from  objectives,  rather  than  from  the  content  of  lessons,  to  insure  that  they 
measure  the  mastery  of  job  objectives,  not  merely  training  content. 

Table  18 

Development  of  Achievement  Tests: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N*=57) 


Prerequisites 

1.  Training  objectives  have  been  specified. 

2.  Instructional  materials  have  not 
been  developed. 

Procedures 

1.  Determine  appropriate  types  of  tests 
based  on  characteristics  of  objectives. 

2.  Construct  tests  to  assess  attainment 
of  objectives. 


Present 

Absent 

Not  Applicable/ 
Not  Determined 

50 

0 

7 

16 

31 

10 

Done 

Not  Done 

Not  Applicable/ 
Not  Determined 

0 

6 

51 

27 

18 

12 

In  27  cases,  tests  were  constructed  to  measure  every  terminal  objective,  and  in  the 
other  cases  testing  was  generally  extensive.  A  widespread  practice  was  the  use  of  criterion- 
referenced  testing  in  place  of  norm-referenced  testing,  a  conversion  that  \  as  often  man¬ 
dated  by  Service  directive.  “Criterion-referenced”  generally  meant  that  scoring  methods 
such  as  the  following  were  employed:  setting  a  relatively  high  percentage  of  correct 
responses  for  passing  a  knowledge  test;  setting  a  low  limit  of  permissible  errors  (often 
zero)  on  performance  tests;  scoring  each  unit  of  a  test  on  an  “all  or  none”  basis;  and 
requiring  trainees  to  pass  all  tests,  retaking  them  as  required. 

Since  Service  policies  discourage  high  failure  rates  from  schools,  the  selection  of 
criteria  obviously  involves  considering  the  norm— what  the  trainees  can  be  expected  to 
learn  in  a  reasonable  time.  For  tests  to  be  strictly  criterion-referenced,  the  criteria  for 
satisfactory  performance  would  be  wholly  determined  by  job  requirements,  and  training 
would  be  increased  as  necessary  to  meet  the  criteria.  In  the  courses  studied,  these  con¬ 
ditions  did  not  prevail. 

The  most  significant  finding  with  regard  to  ISD  application  is  that  tests  were  con¬ 
structed  before  instructional  materials  in  only  one-third  of  the  applicable  cases.  Thus, 
there  was  usually  no  independent  criterion  by  which  to  determine  whether  mastering 
the  course  subject  matter  was  related  to  mastering  the  course  objectives.  This  applied 
especially  to  knowledge  tests,  which  were  hardly  ever  reported  as  having  been  based 
directly  on  training  objectives  (the  objectives  themselves  often  simply  stated  the 
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requirement  to  pass  a  written  test),  thus  leaving  to  those  who  developed  lesson  plans  or 
other  instruction  the  actual  determination  of  what  the  trainee  must  leam. 

It  was  not  determined  in  each  case  whether  test  developers  had  considered  what 
types  of  tests  or  test  items  would  be  appropriate  to  measure  the  objectives.  Many  trainers 
reported  that  objectives  were  classified  as  either  performance  of  knowledge,  and  that  tests 
were  developed  accordingly.  Only  in  rare  cases,  however,  did  personnel  report  they 
used  a  finer  classification  of  objectives  in  developing  test  items  (e.g.,  whether  “behaviors” 
are  skilled  or  unskilled,  whether  “knowledge”  requires  the  application  of  concepts  or  rules 
or  the  recall  of  facts). 
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Identification  of  Entry  Behavior 


When  tests  have  been  developed,  earlier  estimates  of  trainee  entry  behavior  can  be 
verified.  The  derivation  of  objectives  had  been  based  on  these  estimates.  Since  trainees 
are  not  generally  available  at  this  time,  a  determination  of  entry  behavior  is  made  by 
measuring  performance  of  a  group  that  is  representative  of  the  entry  population  in  apti¬ 
tude,  experience,  previous  training,  and  other  relevant  attributes. 

Table  19 


Identification  of  Entry  Behavior: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  <N=57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 
Not  Determined 

1 .  Objectives  have  been  derived  through 

hierarchical  analysis  of  tasks. 

11 

38 

8 

2.  Tests  are  available  to  measure  objectives. 

45 

0 

12 

Procedures 

Done 

Not  Done 

Not  Applicable/ 
Not  Determined 

1.  Identify  sample  that  is  representative 

of  trainees. 

0 

49 

8 

2.  Administer  tests  to  sample,  and 
determine  accuracy  of  earlier  estimate 
of  entry  behavior. 

0 

49 

8 

3.  Add  or  delete  objectives  as  indicated  by 
test  results,  and  repeat  cycle. 

0 

49 

8 

Although  objective-referenced  achievement  tests  were  available  in  many  cases,  and 
hierarchical  analysis  had  been  used  to  some  extent  in  developing  objectives,  no  one 
reported  using  tests  to  verify  the  estimates  or  assumptions  about  trainee  entry  behavior 
underlying  the  objectives.  Unless  estimates  were  always  accurate,  training  requirements 
were  either  overestimated  or  underestimated.  If  trainee  capabilities  were  overestimated, 
the  discrepancy  should  appear  during  validation  of  instruction.  If  trainee  entry  behavior 
had  been  underestimated,  however,  the  resultant  “overtraining”  would  not  necessarily 
be  revealed  by  any  later  ISD  step. 
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Classification  of  Objectives  and 
Selection  of  Instructional  Activities 


Objectives  are  classified  according  to  the  type  of  capability  they  represent,  and 
specific  activities  necessary  to  provide  for  learning  are  identified.  In  later  steps,  media 
will  be  chosen  and  instructional  materials  developed  to  support  these  activities. 

Table  20 

Classification  of  Objectives  and  Selection  of  Instructional  Activities: 
Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N=57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 
Not  Determined 

1 .  Information  is  available  about  types 
of  instructional  activities  appropriate 
to  acquiring  different  types  of  capabilities. 

0 

0 

57 

Procedures 

Done 

Not  Done 

Not  Applicable/ 
Not  Determined 

1.  Classify  each  objective  or  group  of 

objectives  according  to  type  of  capability. 

8 

40 

9 

2.  Specify  instructional  activities  for  each 
objective  according  to  type  of  capability. 

1 

48 

8 

In  only  one  case  were  instructional  activities  reported  to  be  specified  in  accordance 
with  a  distinct  classification  of  behaviors  required  in  objectives  (e.g.,  recall  information, 
perform  gross  motor  movement,  apply  a  rule,  display  an  attitude).  In  some  other  cases, 
objectives  were  classified  in  broad  categories  such  as  motor  skill  or  knowledge.  In  none 
of  these  cases,  however,  were  instructional  activities  specified  in  accordance  with  the 
classification. 

In  a  few  instances,  general  instructional  activities  were  suggested  to  lesson  plan 
writers,  such  as  the  use  of  a  visual  aid  or  the  opportunity  for  trainees  to  practice.  Such 
suggestions,  however,  were  not  reported  as  being  based  on  an  analysis  or  classification 
of  the  characteristics  of  the  objectives. 

As  stated  in  Chapter  2,  the  current  state-of-the-art  of  instructional  technology  does 
not  provide  a  basis  for  a  full  procedure  for  specifying  instructional  activities,  or  even  for 
classifying  objectives.  Also,  the  AFP  50-58  guidance  on  this  subject  appears  to  be 
addressed  only  to  persons  who  will  author  instruction,  not  those  writing  lesson  specifica¬ 
tions.  If  so,  it  would  partly  explain  the  absence  of  such  specifications  in  Air  Force 
ISD  efforts. 

This  does  not  explain  the  almost  universal  failure  of  training  designers  to  carry  out 
this  step  in  the  other  Services,  whose  ISD  models  clearly  recommend  that  instructional 
activities  be  specified.  No  general  explanation  emerged  from  interviews  with  the  training 
designers;  the  step  appeared  rather  to  have  been  ignored.  Possible  explanations  include  the 
novelty  of  the  step;  the  lack  of  a  requirement  to  produce  documents  showing  that  it  was 
carried  out;  incomplete  understanding  of  what  was  meant  by  an  instructional  activity;  and 
reluctance  to  assume  a  function  that  has  formerly  been  the  prerogative  of  instructors  who 
develop  lessons. 


Selection  of  Instructional  Methods 


After  instructional  activities  have  been  specified,  instructional  methods  are  selected. 
These  are  the  ways  in  which  trainees  will  be  brought  into  contact  with  the  instruction. 
Methods  are  selected  to  maximize  the  effectiveness  and  efficiency  of  instruction,  given 
the  particular  instructional  activities,  the  nature  of  the  trainee  population,  the  setting,  and 
administrative  requirements  and  constraints. 

Table  21 

Selection  of  Instructional  Methods: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N=57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 
Not  Determined 

1.  Setting  has  been  specified. 

52 

0 

5 

2.  Trainee  characteristics  have  been  identified. 

0 

1 

56 

3.  Instructional  activities  have  been  specified. 

1 

49 

7 

4.  Information  is  available  on  how  the  costs 
and  effectiveness  of  alternate  methods  vary 
for  specified  settings,  trainee  characteristics, 
and  instructional  activities. 

0 

3 

54 

Procedures 

Done 

Not  Done 

Not  Applicable/ 
Not  Determined 

1 .  Specify  the  methods  of  instruction  to 
be  employed  for  each  objective  or  group 
of  objectives. 

51 

0 

6 

The  determination  of  type  of  pacing,  group  size,  control  of  instruction,  and  other 
instructional  choices  is  inherent  in  implementing  any  training  program.  Thus  Table  21 
shows  that  methods  were  specified  in  all  applicable  courses  (52).  There  was  little  evidence, 
however,  that  the  choice  was  made  on  the  basis  of  the  characteristics  of  the  trainees  or  the 
instructional  activities,  or  with  the  knowledge  of  what  effects  alternate  methods  would 
have  on  training  effectiveness  or  cost.  In  the  typical  case,  it  was  reported  that  no  syste¬ 
matic  procedure  was  used  to  select  methods.  Either  changing  existing  methods  had  not 
been  considered  or  the  option  to  do  so  did  not  rest  with  the  personnel  interviewed. 

In  a  significant  number  of  training  design  efforts,  however,  a  change  in  methods  had 
been  directed,  and  was  the  stated  object  of  the  course  revision.  The  most  common  change 
was  the  conversion  of  courses  to  self -pacing  and  individualization  of  instruction,  notably 
in  the  Army,  and  in  Navy  and  Air  Force  flying  training.  Conversion  to  computer  manage¬ 
ment  of  instruction  is  another  example.  In  both  types  of  cases,  the  decision  was  apparently 
made  by  higher  command  that  a  particular  method  offered  advantages  that  were  general 
enough  to  apply  to  all  training  covered  by  the  directive.  The  basis  on  which  the  decision 
was  made  was  not  determined. 
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Selection  of  Media 


After  instructional  activities  and  training  methods  have  been  specified,  and  before 
instructional  materials  are  developed,  media  are  selected  that  will  provide  an  effective  and 
efficient  means  for  presenting  the  subject  matter  to  the  trainee.  Media  which  satisfy  the 
requirements  of  the  learning  activities  and  methods  are  first  identified,  and  then  costs 
determine  the  final  selection. 


Table  22 

Selection  of  Media: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N-57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 
Not  Determined 

1 .  Training  designers  are  free  to  select 
from  a  range  of  media. 

6 

9 

42 

2.  Instructional  activities  have  been  specified. 

4 

45 

8 

3.  Methods  of  training  have  been  specified. 

39 

1 

17 

4.  Information  is  available  concerning  the 
appropriateness  of  different  media  to  imple¬ 
ment  different  activities  and  to  be  used  in 
conjunction  with  different  methods. 

0 

0 

57 

5.  Information  is  available  concerning  the 
costs  of  different  media. 

0 

0 

57 

Procedures 

Done 

Not  Done 

Not  Applicable/ 
Not  Determined 

1 .  Determine  which  media  will  be  suitable 
to  implement  the  instructional  activities 
and  methods. 

3 

47 

7 

2.  Consider  relative  costs  of  media  deter¬ 
mined  above  and  select  most  economical 
set  of  media. 

1 

48 

8 

The  ISD  procedure  of  identifying  media  appropriate  to  the  instructional  methods  and 
the  stimuli  inherent  in  the  instructional  activities  was  hardly  ever  followed.  Instructional 
activities  had  been  identified  in  only  four  courses.  A  systematic  attempt  to  identify  media 
appropriate  to  particular  objectives  was  made  in  only  three  cases,  and  the  attempt  to  con¬ 
sider  costs  and  select  the  most  efficient  set  of  media  occurred  in  only  one  instance. 

Training  developers  indicated  that  in  general  this  step,  like  the  selection  of  instruc¬ 
tional  methods,  had  simply  not  been  attempted.  Again,  either  no  consideration  was  given 
to  changing  existing  media,  or  training  designers  were  not  at  liberty  to  do  so.  Where 
personnel  interviewed  were  aware  that  this  was  an  ISD  step  they  had  not  carried  out,  the 
cost  of  converting  to  a  different  medium  was  typically  cited  as  a  limitation  on  their 
design  options. 
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Analogous  to  the  selection  of  methods,  converting  to  a  new  medium  was  sometimes 
the  primary  reason  for  revising  a  course.  In  such  a  case  media  selection  occurred  before 
any  of  the  other  ISD  steps.  Such  conversions  were  typically  undertaken  to  facilitate  a 
change  to  individualized  instruction. 

As  noted  in  Chapter  2,  the  current  state  of  instructional  technology  does  not  permit 
the  prediction  of  which  medium  will  be  most  effective  in  training  for  a  given  objective. 
The  algorithms  provided  in  the  various  ISD  manuals  are  not  sufficient  by  themselves  to 
allow  a  training  designer  to  identify  the  most  efficient  medium,  even  if  cost  information 
were  available  and  the  stimulus  requirements  of  instructional  activities  could  be  identified 
in  every  instance.  Whether  training  designers  perceived  the  inadequacy  of  the  procedural 
aids  is  not  clear.  In  any  event,  no  attempt  to  use  them  was  reported. 
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Grouping  and  Sequencing  of  Instruction 

Objectives  are  grouped  and  sequenced  acco  ding  to  dependency  and  commonality 
of  subject  matter,  and  transfer  and  efficiency  oi  learning.  Decisions  are  also  subject 
to  practical  and  administrative  considerations.  A  principal  characteristic  of  ISD  is  that 
decisions  about  sequencing  and  grouping  be  made  on  the  basis  of  learning  factors  as  much 
as  possible,  rather  than  solely  on  the  basis  of  administrative  factors  or  past  practice. 

Table  23 

Grouping  and  Sequencing  of  Instruction: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N=57) 


Prerequisites 

1 .  Knowledge  of  the  effects  on  learning 
of  different  sequencing  plans  is  available. 


Procedures 

1.  Identify  commonality  of  subject  matter 
and  anticipated  transfer  of  learning 
between  objectives. 

2.  Identify  degree  of  dependency  between 
objectives. 

3.  Select  overall  sequencing  principle(s). 

4.  Group  and  sequence  objectives. 


Not  Applicable/ 
Not  Determined 


Not  Applicable/ 
Not  Done  Not  Determined 


7 

39 

11 

13 

29 

15 

13 

32 

12 

51 

0 

6 

As  with  selection  of  methods,  a  determination  of  sequence  is  inherent  in  any  training 
program,  and  Table  23  shows  accordingly  that  this  was  accomplished  in  all  applicable 
cases.  However,  a  systematic  attempt  to  base  the  sequence  of  instruction  on  the  degree 
of  commonality,  dependency,  or  learning  transfer  among  objectives  was  reported  in  only 
six  cases.  It  may  be  significant  that  two  of  these  six  cases  were  Navy  flying  training  courses 
developed  by  civilian  contractors.  Although  commonality  or  dependency  was  identified 
in  some  other  cases  as  well,  there  was  no  evidence  that  these  relationships  among  objec¬ 
tives  had  been  used  to  determine  the  most  effective  instructional  sequence. 

In  the  typical  case,  no  consideration  was  given  to  how  alternate  sequencing  strategies 
would  affect  learning.  In  many  cases  the  schedule  of  equipment  and  facilities  availability 
was  cited  as  the  controlling  factor  in  sequencing.  In  other  cases,  it  was  reported  that 
the  proper  sequence  was  obvious  or  had  followed  “logical”  order,  which  generally  meant 
providing  instruction  on  one  system  or  piece  of  equipment  at  a  time,  beginning  with  theory 
of  operation  or  other  general  information  and  proceeding  to  specific  information  or  practice. 
If  an  existing  course  was  revised,  changing  the  existing  sequence  was  usually  not  considered. 

Next  to  equipment  and  facilities  availability,  dependency  is  perhaps  the  most  readily 
apparent  sequencing  consideration.  Factors  such  as  transfer  of  learning  are  harder  to  esti¬ 
mate,  and,  as  noted  in  Chapter  2,  principles  of  sequencing  are  ambiguous.  It  may  be  that 
training  developers  did  not  consider  that  a  systematic  procedure  was  necessary  to  discover 
what  was  dependent,  and  that  any  attempt  to  consider  other  factors  offered  little  promise. 
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Development  of  Plan  for 
Authoring  and  Managing  Instruction 


A  plan  is  prepared  to  guide  those  who  will  author,  conduct,  and  manage  the  instruc¬ 
tional  program.  Whether  in  a  single  master  plan  or  in  separate  documents  intended  for 
the  different  personnel  concerned,  the  decisions  and  specifications  made  in  previous  ISD 
steps  are  transmitted  to  those  who  will  carry  them  out.  Decisions  about  administrative 
matters  such  as  class  schedules  and  equipment  use  are  included. 


Table  24 

Development  of  Plan  for  Authoring  and  Managing  Instruction: 
Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  IN=57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 
Not  Determined 

1.  Objectives,  instructional  activities, 
methods,  and  media  have  all  been 
specified. 

5 

44 

8 

Procedures 

Done 

Not  Done 

Not  Applicable/ 
Not  Determined 

1 .  Specify  the  content  and  design  of 

each  lesson. 

31 

15 

11 

2.  Specify  how  the  instruction  will  be 

conducted  and  managed. 

50 

1 

6 

In  31  courses,  the  design  and  content  of  each  lesson  were  specified  in  a  plan  of  some 
kind.  The  extent  to  which  such  plans  had  been  developed  prior  to  authoring  instruction 
and  had  guided  lesson  authors  could  not  be  determined.  Generally,  there  was  little  evidence 
that  they  had.  A  typical  plan  was  a  Program  of  Instruction  or  Curriculum  Outline  listing 
lessons,  objectives,  number  of  hours  for  training,  equipment  required,  etc. 

While  objectives,  instructional  methods,  and  media  were  available  in  nearly  all  cases 
in  which  plans  had  been  prepared,  instructional  activities  had  been  identified  in  less  than 
10%  of  all  courses  examined.  Thus,  the  extent  to  which  specific  activities  were  included 
in  authoring  plans  was  necessarily  quite  limited.  The  plans  available  in  the  courses  studied 
appeared  to  be  intended  more  to  record  the  content  and  control  the  conduct  of  training 
than  to  provide  design  specifications  for  lesson  authors.  In  virtually  all  cases,  plans 
included  information  about  how  instruction  would  be  conducted  and  managed  (e.g.,  group 
or  self-paced,  instructor  or  computer  managed). 


86 


/ 


Review  and  Selection  of  Existing  Materials 

After  instructional  activities  and  media  have  been  specified,  any  available  existing 
instructional  materials  are  examined  to  see  if  they  match  these  specifications  in  whole  or 
in  part.  Materials  that  match  can  be  used,  with  resultant  savings  in  the  authoring  of 
instruction.  In  addition  to  satisfying  established  specifications  for  instructional  activities, 
methods,  and  media,  materials  adopted  must  also  be  appropriate  to  trainee  characteristics 
and  the  instructional  management  plan  of  the  proposed  course. 

Table  25 


Review  and  Selection  of  Existing  Materials: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N=57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 

Not  Determined 

1.  Instructional  activities  have  been  specified. 

5 

43 

9 

2.  Methods  of  training  have  been  specified. 

49 

0 

8 

3.  Media  have  been  specified. 

37 

7 

13 

Procedures 

Done 

Not  Done 

Not  Applicable/ 

Not  Determined 

1.  Examine  existing  instructional  materials 
to  determine  whether  any  meet  the 
specifications  for  instructional  activities, 
methods,  and  media. 

10 

36 

ii 

2.  Select  materials  or  parts  of  materials  that 
meet  the  specifications,  or  which  could  be 
efficiently  revised." 

11 

35 

ii 

Existing  materials  were  reviewed  and  selected  in  10  courses.  In  most  cases,  the 
materials  under  review  were  simply  those  of  the  course  being  revised.  Instructional 
materials  from  other  Services  or  the  civilian  sector  were  reviewed  in  only  a  few  instances. 

Characteristics  of  the  planned  instruction,  such  as  methods,  media,  and  instructional 
activities,  rarely  provided  the  basis  for  the  review  and  selection  process.  (Methods  and 
media  specifications  were  available  in  most  of  the  applicable  cases,  but  instructional 
activities  in  only  five.)  Rather,  the  typical  method  was  to  inspect  materials  on  the  basis 
of  general  suitability  to  the  new  or  revised  course. 
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Authoring  of  Instruction 


The  content  of  instruction  is  determined  by  earlier  decisions  with  respect  to  grouping 
and  sequencing  of  objectives,  instructional  activities,  methods,  media,  and  the  specification 
of  content  for  individual  units/lessons  in  the  authoring  plan.  The  actual  authoring  process 
consists  of  preparing  scripts,  texts,  tape/slide  programs,  etc.  As  first  drafts  of  instructional 
materials  are  produced,  small  portions  are  tried  out  on  individuals  who  are  representative 
of  the  entry  population.  Materials  are  then  revised  to  correct  weaknesses,  omissions,  and 
ambiguities.  A  distinguishing  characteristic  of  ISD  is  initially  to  include  only  the  bare 
minimum  of  instruction  and  subsequently  to  augment  instruction  as  needed. 

Table  26 


Authoring  of  Instruction: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N=57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 

Not  Determined 

1 .  Objectives  have  been  grouped  and 
sequenced. 

49 

0 

8 

2.  Instructional  activities  have  been 
specified. 

5 

42 

10 

3.  Methods  of  training  have  been  specified. 

48 

0 

9 

4.  Media  have  been  specified. 

37 

8 

12 

5.  Lesson  structure  and  content  have 
been  planned. 

30 

14 

13 

Procedures 

Done 

Not  Done 

Not  Applicable/ 

Not  Determined 

1.  Develop  lean  instruction. 

11 

28 

18 

2.  Try  out  instruction  on  small  number  of 
persons  representative  of  students. 

10 

31 

16 

3.  Revise  and  augment  instruction  as 
necessary. 

5 

30 

22 

As  indicated  in  Table  26,  the  sequence  and  methods  of  instruction  were  nearly  always 
determined  prior  to  authoring  materials,  while  the  choice  of  media  was  occasionally  left 
to  lesson  authors.  By  contrast,  instructional  activities  were  specified  only  about  10%  of 
the  time.  A  plan  for  lesson  authors  was  available  in  about  two-thirds  of  the  cases. 

In  only  11  courses  was  there  an  attempt  to  restrict  the  content  of  instruction  to  the 
minimum  necessary.  Most  interviewees  were  not  familiar  with  this  concept.  Although 
other  ISD  steps  were  often  reported  to  have  been  omitted  due  to  constraints  of  time  or 
money,  lack  of  awareness  appears  to  be  the  important  factor  with  respect  to  developing 
lean  instruction.  Anyone  who  reported  an  awareness  of  the  advantages  of  keeping  instruc¬ 
tion  lean  also  reported  trying  to  do  so.  By  contrast,  some  interviewees  reported  that  they 
had  done  just  the  opposite— with  no  apparent  awareness  that  their  approach  was  contrary 
to  the  general  intent  of  ISD. 
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Nine  of  the  11  cases  of  lean  instruction  were  found  in  the  Navy.  These  include 
both  courses  developed  by  ISD  contractors  for  the  Naval  Training  Equipment  Center, 
and  both  courses  developed  by  the  Instructional  Program  Development  Center,  San 
Diego— presumably  reflecting  a  greater  awareness  of  the  requirements  and  purpose  of  ISD 
among  those  permanently  engaged  in  course  design.  Except  in  these  cases,  the  virtual 
restriction  to  the  Navy  of  efforts  to  keep  instruction  lean  is  unexplained. 

The  development  of  lean  instruction  was  found,  not  surprisingly,  to  be  associated 
with  efforts  to  try  out  and  revise  instruction,  and  with  the  subsequent  ISD  step  of 
attempting  to  validate  it.  Although  instructional  materials  were  tried  out  and  revised  in 
only  five  cases,  three  of  the  five  represent  instances  where  there  was  an  attempt  to  provide 
lean  instruction.  Instruction  was  validated  in  less  than  one-fourth  of  the  total  applicable 
courses,  but  was  valided  in  two-thirds  of  the  instances  of  lean  instruction.  The  association 
of  tryout,  revision,  and  validation  with  lean  instructional  development  is  quite  reasonable 
since  the  practice  of  restricting  instructional  content  to  a  minimum  requires  the  use  of 
some  parallel  procedure  to  measure  instructional  adequacy. 
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Validation  of  Instruction 

Instruction  is  tried  out  under  conditions  that  closely  approximate  its  intended  use, 
with  groups  of  trainees  representative  of  the  entry  population.  Objective-referenced 
achievement  tests  are  the  primary  validation  criterion. 


Table  27 

Validation  of  Instruction: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N=57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 

Not  Determined 

1 .  Objective-referenced  achievement  tests 
are  available. 

47 

0 

10 

Procedures 

Done 

Not  Done 

Not  Applicable/ 

Not  Determined 

1 .  Specify  achievement  test  validation 
criteria  (number  and  percent  of  per¬ 
sons  in  validation  sample  required  to 
pass  tests). 

7 

30 

20 

2.  Specify  additional  validation  criteria. 

4 

32 

21 

3.  Present  instruction,  administer  achieve¬ 
ment  tests,  analyze  results,  revise  instruc¬ 
tion,  and  repeat  cycle  until  validation 
criteria  are  met. 

9 

31 

17 

Although  objective-referenced  achievement  tests  were  available  in  every  case,  instruc¬ 
tional  materials  were  subjected  to  a  validation  process  in  only  nine  courses.  Even  in  two 
of  these  nine  cases,  no  criteria  of  adequacy  were  established  for  the  validation  trials.  Seven 
of  the  nine  validation  efforts  occurred  in  the  Navy,  including— as  in  the  case  of  developing 
lean  instruction— both  courses  at  IPDC  San  Diego  and  a  contractor-developed  course  for 
NTEC  (the  other  NTEC-sponsored  contractor  effort  had  not  reached  the  point  of  valida¬ 
tion  at  the  time  of  the  interview). 

For  many  courses,  no  self-teaching  materials  (tape/slide,  text,  etc.)  were  being 
produced— only  lesson  plans  to  guide  instructors’  presentations.  In  such  cases,  the  vali¬ 
dation  process  would  require  repeated  presentation  of  the  lesson  under  controlled  condi¬ 
tions.  No  one,  however,  reported  following  such  a  procedure;  there  were  no  efforts  to 
validate  lesson  plans. 

Interviewees  at  Army  schools  reported  that  the  TEC  lessons  being  produced  to 
support  individual  training  are  subjected  to  a  validation  process.  No  TEC  lessons  were 
reported  to  be  used  in  the  courses  studied,  and  consequently  no  TEC  development  efforts 
were  reviewed,  but  at  least  one  instance  of  such  validation  was  briefly  observed. 


90 


J 


Internal  Evaluation 

Internal  evaluation  determines  how  well  instruction  works  during  actual  training  with 
actual  trainees.  Validation  occurs  only  once,  while  internal  evaluation  is  continuous.  The 
principal  measure  is  trainee  performance  on  objective-referenced  achievement  tests.  Anal¬ 
ysis  or  evaluation  of  how  the  ISD  process  itself  was  carried  out  is  also  part  of  internal 
evaluation,  and  is  sometimes  termed  process  evaluation. 

Table  28 

Internal  Evaluation: 

Incidence  of  Prerequisites  and  Procedures  ir.  Courses  Examined  (N=57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 
Not  Determined 

1.  Records  of  students' performance  on 
achievement  tests  are  available. 

28 

1 

28 

2.  Documentation  is  available  of  what 
occurred  during  the  ISD  process, 
including  rationales  for  decisions, 
departures  from  standard  procedures,  etc. 

13 

19 

25 

Procedures 

Done 

Not  Done 

Not  Applicable/ 
Not  Determined 

1.  Specify  evaluation  criteria  (number  and 
percent  of  persons  required  to  pass 
tests,  etc.). 

4 

28 

25 

2.  Identify  the  causes  of  shortcomings  in  the 
instruction  and  specify  revisions. 

15 

18 

24 

t 

i 

Efforts  to  investigate  shortcomings  in  ongoing  instruction  and  recommend  remedies 
were  reported  in  15  courses.  Such  efforts  rarely  involved  comparing  trainee  performance 
to  a  standard  of  course  adequacy;  despite  the  almost  universal  availability  of  achievement 
test  scores,  a  criterion  of  course  adequacy  was  established  in  only  four  cases.  Possibly 
trainers  see  little  need  for  such  standards  in  light  of  established  limits  on  course  attrition. 
If  a  requirement  for  course  completion  is  to  achieve  every  objective,  for  example,  and 
if  permissible  course  attrition  rates  are  determined  by  higher  command,  it  may  be  felt 
that  no  other  standards  are  necessary. 

Some  documentation  of  the  ISD  process  itself  was  available  in  13  courses.  The  com¬ 
prehensiveness  of  such  documentation  varied  greatly.  In  some  cases  records  were  available 
showing  when  different  activities  had  been  started  and  completed,  but  in  most  instances 
documentation  was  less  detailed.  Notably  absent  from  nearly  all  documentation  were 
rationales  for  the  selection  of  tasks,  setting,  methods,  and  media. 


External  Evaluation 


Trainee  proficiency  and  the  adequacy  of  the  instructional  design  process  are  deter¬ 
mined  by  using  a  standard  external  to  the  course:  the  performance  of  the  graduate  on 
the  job.  The  most  accurate  methods  of  external  evaluation  are  direct  observation  and 
testing  of  graduates.  Reliance,  however,  must  usually  be  placed  on  supervisors’  summary 
evaluations  (ratings)  of  performance.  Evaluation  information  of  whatever  type  should  be 
obtained  at  a  task  level  of  specificity.  More  general  evaluations  are  of  little  use  in  iso¬ 
lating  the  causes  of  inadequate  performance. 

Table  29 


External  Evaluation: 

Incidence  of  Prerequisites  and  Procedures  in  Courses  Examined  (N=57) 


Prerequisites 

Present 

Absent 

Not  Applicable/ 

Not  Determined 

1.  Access  to  supervisors  and  job  incumbents 
is  available  soon  after  arrival  of  graduates 
on  the  job. 

24 

1 

32 

Procedures 

□one 

Not  Done 

Not  Applicable/ 

Not  Determined 

1.  Construct  evaluation  instruments  (mail 
questionnaires,  job  sample  tests,  inter¬ 
view  guides,  etc.). 

19 

15 

23 

2.  Collect  evaluation  information. 

16 

15 

26 

3.  Analyze  data,  identify  causes  of 
deficiencies,  and  specify  revisions. 

8 

17 

32 

Information  about  graduates’  performance  in  the  field  was  collected  in  16  courses. 

In  half  of  these  cases,  the  information  was  analyzed  and  used  in  making  some  revision  in 
the  course.  Questionnaires  to  graduates  and  supervisors— usually,  but  not  always,  written 
at  the  task  level— were  the  main  method  employed.  In  one  instance  a  pre-existing  Job 
Performance  Measure  was  used. 

In  most  of  these  cases  the  persons  responsible  for  course  development  also  planned 
and  conducted  the  external  evaluation.  In  two  courses,  the  Training  Evaluation  Division 
of  an  Air  Force  technical  training  school  conducted  the  evaluation  and  issued  a  formal 
Training  Evaluation  Report.  Formal  evaluations  of  the  same  kind  were  conducted  for 
some  Army  courses  as  well,  by  the  school  Directorate  of  Evaluation.  Although  no  exter¬ 
nal  evaluations  had  been  completed  for  any  of  the  Army  courses  studied,  two  were 
scheduled  to  begin  in  the  near  future. 

In  many  of  the  cases  in  which  graduates’  opinions  or  information  about  their  perform¬ 
ance  was  collected,  however,  the  effort  did  not  appear  to  have  a  clear  connection  with  the 
instructional  design/revision  process.  Some  interviewees  reported  that  data  collected  from 
graduates  or  supervisors  had  never  been  made  available  to  those  who  had  designed  the  course. 
In  other  cases  periodic  surveys  were  conducted,  as  directives  required,  but  with  no  apparent 
idea  of  how  the  survey  responses  would  be  used  to  determine  the  need  to  revise  instruction. 
In  general,  whether  personnel  could  perform  on  the  job  the  tasks  for  which  they  had  been 
trained  was  not  determined.  The  ISD  “closed  loop”  remained  open. 


SUMMATION 


The  great  majority  of  ISD  steps  are  either  rarely  undertaken  in  the  Services  or  are 
carried  out  in  a  manner  which  fails  to  meet  the  requirements  of  ISD .  Application  of 
ISD  in  any  real  sense  is  virtually  restricted  to  five  steps:  Identification  of  Job  Require¬ 
ments,  Analysis  of  Tasks,  Development  of  Objectives  and  Objectives  Hierarchies,  Devel¬ 
opment  of  Achievement  Tests,  and  External  Evaluation.  These  are  all  steps  for  which 
procedures  are  available,  and  which  do  not  involve  considerations  of  total  system  per¬ 
formance  or  training  subsystem  efficiency. 

The  execution  of  even  these  steps,  however,  often  falls  short  of  achieving  the  intent 
and  purpose  of  ISD.  Job  requirements  are  often  defined  simultaneously  with,  rather  than 
prior  to,  the  determination  of  training  requirements,  thus  inviting  training  considerations 
to  influence  job  analysis;  the  means  by  which  job  requirements  then  become  training 
requirements  are  generally  not  explicit.  Task  analysis  does  not  usually  include  the  delinea¬ 
tion  of  task  conditions  and  standards,  on  which  the  derivation  of  training  conditions  and 
standards  would  logically  depend.  Objectives  are  as  likely  to  be  developed  from  instruc¬ 
tional  content,  as  content  from  objectives;  objectives  hierarchies  per  se  are  rarely  developed, 
with  the  result  that  necessary  basic  skills  and  knowledge  are  usually  not  incorporated  in 
training  design  specifications.  Tests  are  often  derived  from  instructional  content  rather 
than  directly  from  objectives,  raising  the  possibility  that  such  tests  may  fail  to  measure 
attainment  of  the  actual  objectives.  Information  collected  about  graduates’  job  perform¬ 
ance  is  seldom  used  to  revise  training. 

The  potential  of  ISD  to  insure  that  training  meets  job  requirements  depends  both 
upon  (a)  maintaining  the  interdependency  of  steps  in  a  derivative  process  originating 
in  description  of  the  job  itself,  and  (b)  testing  and  revising  the  products  of  each  of  these 
steps  until  they  meet  previously  identified  requirements.  In  practice,  however,  the  nec¬ 
essary  close  connection  between  steps  is  generally  not  maintained,  nor  is  the  cycle  of 
testing  and  revision  carried  through. 

The  principal  effect  of  the  current  emphasis  on  ISD  appears  to  be  an  awareness  of 
contemporary  training  methods  and  techniques.  The  use  of  behavioral  objectives  to 
describe  training  content  and  the  specification  of  criteria  of  mastery  for  training  objectives 
is  almost  universal;  and  there  is  widespread  awareness  of  the  desirability  of  restricting 
training  to  what  is  needed  for  job  performance.  Although  the  ISD  cycle  of  evaluation 
and  revision  is  hardly  ever  carried  through  successive  iterations,  there  is  widespread  accept¬ 
ance  of  the  principle  that  trainees’  performance  is  the  valid  measure  of  training  effective¬ 
ness.  Although  instructional  activities,  sequence,  methods,  and  media  are  still  determined 
by  ad  hoc  personal  judgment  rather  than  systematic  procedures,  the  conversion  of  tradi¬ 
tional  group-paced  training  to  individualized  self-paced  instruction,  and  the  replacement 
of  the  lecture  or  tutorial  with  other  media,  are  common  occurrences  in  ISD  efforts. 
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Chapter  6 

FINDINGS  AND  RECOMMENDATIONS 


The  present  study  sought  answers  to  these  questions: 

•  Do  current  methodologies  as  represented  in  guidance  documents  provide 
the  means  for  attaining  the  goals  of  Instructional  System  Development? 

•  Do  applications  of  ISD  reflect  these  goals? 

•  How  can  ISD  methodologies  and  applications  be  made  more  effective? 

The  basic  information  for  answering  these  questions  was  sought  by  analyzing  guidance 
for  19  individual  steps  of  ISD  in  four  Service  Models,  and  the  manner  in  which  each  step 
was  carried  out  in  57  course  design  efforts.  This  approach  yielded  findings  about  meth¬ 
odologies  (Chapter  2)  and  their  applications  (Chapters  4  and  5)  specific  to  particular 
steps,  as  well  as  more  general  findings  about  the  system.  In  this  chapter  findings  and 
accompanying  recommendations  for  specific  steps  are  presented  first.  They  are  followed 
by  a  discussion  of  the  general  adequacy  of  ISD  and  recommendations  for  its  future 
implementation. 


FINDINGS  AND  RECOMMENDATIONS  FOR 
SPECIFIC  ISD  STEPS 


Need/Discrepancy  Analysis 


Findings 

1.  Methodology  -  Present  ISD  models  do  not  include  procedures  for  identifying  problems 
in  existing  training  and  for  identifying  a  need  to  undertake  ISD.  Guidance  is  needed 
on  how  to  identify  discrepancies  between  existing  training  and  the  field  requirements 
for  a  job,  and  how  to  revise  training  short  of  undertaking  the  entire  ISD  process. 

2.  Application  -  ISD  is  not  generally  initiated  in  response  to  specific  discrepancies 
between  training  and  field  requirements. 

3.  Application  -  ISD  is  usually  initiated  in  response  to  a  directed  change  (e.g., 
provide  individualized  and  self-paced  instruction)  or  to  a  requirement  to  revise 
existing  training  in  accordance  with  ISD  methods. 

Recommendations 

1.  Where  existing  training  is  being  examined,  ISD  methodologies  should  emphasize 
evaluating  and  improving  the  training,  rather  than  simply  assuming  that  development 
of  a  new  course  is  appropriate.  At  present  new  courses  are  rarely  evaluated  any 
more  rigorously  than  the  ones  they  replace. 

2.  To  increase  the  emphasis  on  evaluation  and  improvement,  specific  procedures  should 
be  developed  both  to  identify  faults  in  existing  training  and  to  determine  efficient 
boundaries  for  the  ISD  process. 
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IDENTIFICATION  OF  JOB  REQUIREMENTS 


Findings 

1.  Methodology  -  Current  guidance  for  identifying  job  requirements  permits  con¬ 
siderable  latitude  in  the  approach  taken  and  the  level  of  description  used,  with 
resulting  variation  in  the  reliability  and  utility  of  the  information  developed. 

2.  Application  -  Job  analysis  is  usually  confounded  with  the  selection  of  tasks  for 
training.  Emphasis  is  not  given  to  independently  specifying  requirements  as  they 
exist  in  the  job. 

3.  Application  -  Job  task  lists  from  occupational  surveys  (e.g.,  CODAP,  NOTAP)  are 
sometimes  available.  The  information  provided  by  such  lists,  however,  is  often  in 
part  about  classes  of  activities  rather  than  about  tasks,  and  to  that  extent  may  not 
serve  as  an  adequate  base  for  deriving  training.  Often,  complete  task  lists  are  not 
developed. 

Recommendation 

1.  Training  developers  should  be  required  to  provide  and  maintain  a  description  of  job 
task  requirements  distinct  from  a  listing  of  tasks  selected  for  training.  This  would 
make  explicit  the  extent  to  which  training  requirements  differ  from  job  requirements. 


Selection  of  Tasks  for  Training 


Findings 

1.  Methodology  -  There  are  no  measures  of  system  effectiveness  that  can  be  used  to 
validate  criteria  for  selecting  tasks  for  training,  and  rules  for  applying  such  criteria. 

As  a  result,  the  choice  of  criteria  to  be  used  in  selecting  tasks  to  be  trained  is  left 
to  persona]  judgment. 

2.  Application  -  Selecting  tasks  for  training  is  generally  not  preceded  by  a  separate 
and  distinct  delineation  of  the  tasks  required  by  the  job. 

3.  Application  -  Task  selection  for  training  is  usually  not  done  systematically.  Its 
rationale  is  rarely  explicit. 

4.  Application  -  Task  priorities  (that  would  provide  the  basis  for  getting  the  maximum 
training  benefits  from  the  available  funding)  are  not  specified. 

Recommendations 

1.  Training  developers  should  be  required  to  make  explicit  the  basis  on  which  they 
select  tasks  for  training,  and  to  specify  priorities  among  both  the  tasks  selected  and 
the  tasks  rejected  for  training  for  a  particular  job. 

2.  In  the  absence  of  information  about  the  effects  of  task  selection  criteria  on  system 
performance,  guidance  should  be  developed  on  the  types  o£»task  priority  information 
likely  to  be  relevant  for  different  classes  of  jobs. 


Analysis  of  Tasks 


Findings 

1.  Methodology  -  Procedures  for  analyzing  tasks  are  adequate. 

2.  Application  -  Job  task  conditions  and  standards,  as  distinct  from  training  conditions 
and  standards,  are  seldom  identified.  Course  control  documents  do  not  require  that 
job  characteristics  be  specified. 

Recommendation 

1.  Benefits  of  attempting  to  modify  current  practices  do  not  appear  great  relative  to 
costs.  No  change  in  present  practice  is  recommended. 


Construction  of  Job  Performance  Measures 


Findings 

1.  Methodology  -  Procedures  for  developing  JPMs  are  adequate  in  the  1TRO  model, 
unclear  in  the  Marine  Corps  model,  and  not  included  in  the  AFP  50-58  model. 

2.  Application  -  JPMs  have  been  developed  in  jobs  where  the  consequences  of  inade¬ 
quate  performance  are  especially  serious,  such  as  in  flight  training  or  use  of  special 
weapons. 

3.  Application  -  In  only  one  instance  were  JPMs  developed  as  part  of  ISD  to  validate 
within-course  tests. 

Recommendation 

1.  While  theoretically  worthwhile,  JPMs  are  costly  to  develop.  It  is  unrealistic  to 
recommend  their  development  except  in  special  instances.  No  change  in  present 
practice  is  recommended. 


Selection  of  Setting 


Findings 

1.  Methodology  -  A  systematic  procedure  for  determining  the  optimal  setting  for 
training  does  not  exist.  The  development  of  such  a  procedure  waits  upon  a  means 
for  measuring  system  performance,  which  in  turn  will  permit  the  validation  of  site 
selection  criteria  and  decision  rules. 

2.  Application  -  In  general,  training  developers  do  not  have  the  authority  to  designate 
and  develop  training  in  different  settings. 

Recommendation 

1.  The  choice  of  training  setting  has  effects  on  the  operational  system  beyond  the 
training  subsystem.  It  is  appropriate  that  the  setting  selection  be  made  at  a  higher 
level  than  training  developer.  In  the  absence  of  a  means  for  assessing  total  system 
effectiveness,  no  change  in  present  practice  is  recommended. 
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Development  of  Training 
Objectives  and  Objectives  Hierarchies 


Findings 

1.  Methodology  -  The  ITRO  model  provides  the  most  comprehensive,  explicit,  and 
straightforward  procedures  for  translating  job  requirements  into  training  objectives. 
Deficiencies  in  the  Marine  Corps  model  could  encourage  writing  objectives  to  match 
what  is  being  taught,  rather  than  to  meet  job  requirements. 

2.  Application  -  The  specification  of  training  objectives  is  virtually  universal,  but  the 
procedures  used  to  identify  objectives  are  highly  variable  and  frequently  unclear. 
There  is  evidence  that  objectives  are  often  prepared  after  the  fact  and  are  derived 
from  training  content  rather  than  used  to  generate  it. 

3.  Application  -  Terminal  objectives  are  seldom  subjected  to  an  explicit  hierarchical 
analysis  to  derive  intermediate  objectives.  A  determination  of  the  skills  and  knowl¬ 
edge  that  would  enable  the  trainee  to  meet  the  terminal  objectives  would  emerge 
from  such  an  analysis;  these  often  are  not  identified. 

4.  Application  -  Even  where  records  are  maintained,  formats  for  displaying  the  relation 
between  tasks  and  training  objectives  make  it  hard  to  determine  what  objectives 
have  been  derived  from  a  given  task.  That  is,  tasks  that  represent  objectives  are 
displayed,  rather  than  objectives  that  have  been  derived  from  each  task.  Thus,  the 
justification  for  training  for  specific  objectives  is  often  not  clear. 

Recommendation 

1.  The  derivation  of  training  objectives  from  job  tasks  should  be  made  explicit  in  a 
format  that  cross-references  objectives  by  task. 


Development  of  Achievement  Tests 
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Findings 

1.  Methodology  -  All  of  the  models  require  that  achievement  tests  be  developed  from 
training  objectives,  rather  than  from  the  content  of  lessons,  and  all  provide  some 
information  about  test  construction.  All  models  lack  procedures  for  maintaining 
congruence  between  the  behaviors  implied  in  an  objective  and  the  actual  require¬ 
ments  imposed  by  test  items  (e.g.,  use  performance  tests  to  measure  skilled  behavior; 
require  that  concepts  be  applied  when  an  objective  implies  their  use  rather  than 
their  recall  or  recognition). 

2.  Application  -  Many  achievement  tests  are  derived,  not  directly  from  training  objec¬ 
tives,  but  from  training  content.  Knowledge  tests  are  particularly  likely  to  be 
derived  from  content.  In  these  cases,  no  independent  criterion  exists  to  deter¬ 
mine  whether  training  objectives  have  been  met. 

3.  Application  -  In  general,  little  or  no  consideration  is  given  to  matching  the  type  and 
level  of  test  items  to  the  behavioral  requirements  of  objectives  (see  Finding  1). 

Recommendations 

1.  IS D  models  should  explain  and  emphasize  the  purpose  and  need  for  deriving 
achievement  tests  from  training  objectives  rather  than  from  training  content. 

2.  ISD  models  should  be  expanded  to  provide  procedures  for  identifying  and  main¬ 
taining  congruence  between  the  behavioral  requirements  of  objectives  and  test  items. 
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Findings 

1.  Methodology  -  The  major  ISD  models  provide  procedures  for  adjusting  training 
objectives  to  match  trainee  capabilities. 

2.  Application  -  Estimates  of  capabilities  of  trainee  populations  are  not  verified  by 
testing  before  the  training  is  implemented. 

Recommendation 

1.  The  advantages  of  correcting  inaccurate  estimates  of  trainee  capabilities  do  not 
appear  to  justify  the  costs  of  the  measurement  that  would  be  required.  Over¬ 
estimates  are  likely  to  be  identified  during  the  validation  of  instruction,  and  under¬ 
estimates  usually  become  evident  during  the  conduct  of  instruction. .  No  change  in 
present  practice  is  recommended  unless  large  investments  in  instructional  materials 
are  involved  (e.g..  Training  Extension  Course  (TEC)  development). 


Classification  of  Objectives  and 
Selection  of  Instructional  Activities 


Findings 

1.  Methodology  -  Procedures  for  classifying  training  objectives,  and  for  selecting  instruc¬ 
tional  activities  accordingly,  are  not  highly  developed.  Different  models  use  different 
taxonomies  for  classifying  objectives,  and  guidance  for  both  classifying  objectives 
and  selecting  instructional  activities  is  provided  largely  by  example  rather  than  by 
means  of  explicit  decision  rules. 

2.  Application  -  Training  objectives  generally  are  not  classified,  and  instructional  activ- 
ities  generally  are  not  specified. 

Recommendation 

1.  Explicit  decision  rules  for  classifying  objectives  and  selecting  instructional  activities 
should  be  developed.  In  the  absence  of  such  rules,  no  change  in  present  practice 
is  recommended. 


Selection  of  Instructional  Methods 


Findings 

1.  Methodology  -  All  models  describe  ar  d  discuss  alternative  instructional  methods. 

They  specify  prerequisite  conditions  (e.g.,  setting,  group  size)  for  the  use  of  particu¬ 
lar  methods  but  provide  little  information  about  the  relative  effectiveness  of  different 
methods,  either  for  particular  types  of  content  or  for  trainee  populations.  Though 
perhaps  sufficient  to  allow  the  training  developer  to  reject  inappropriate  methods, 
the  information  base  and  the  models  themselves  are  not  sufficient  to  provide  for 
selecting  optimal  methods. 
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2.  Application  -  Training  methods  are  not  systematically  selected  either  on  th^basis 
of  instructional  activities  (which  are  also  not  specified)  or  on  the  basis  of  trainee 
characteristics. 

3.  Application  -  Changes  in  training  methods  are  almost  always  in  response  to  com¬ 
mand  policy. 

Recommendation 

1.  Present  training  technology  is  not  advanced  enough  to  support  proceduralized 
derivation  of  training  methods  from  previously  specified  instructional  activities 
and  trainee  characteristics.  Information  about  optimal  training  methods  for  dif¬ 
ferent  training  situations,  and  procedures  to  enable  developers  to  identify  the  moq|, 
promising  methods,  should  be  developed.  In  the  absence  of  such  procedures,  no 
change  in  present  practice  is  recommended. 

Selection  of  Media 

Findings 

1.  Methodology  -  Procedures  for  selecting  training  media  appear  adequate  to  match 
the  presentation  (stimulus  and  response)  requirements  of  instructional  activities  to 
appropriate  media,  if  instructional  activities  have  been  specified  in  sufficient  detail. 

2.  Application  -  Training  media  are  not  systematically  selected  on  the  basis  of  require¬ 
ments  of  instructional  activities  (which  are  also  not  specified). 

3.  Application  -  Developers  generally  do  not  have  freedom  to  select  among  alternative 
media.  Choices  and  changes  in  media  are  usually  directed  by  command  policy. 

Recommendation 

1.  Selection  of  appropriate  media  is  contingent  on  how  well  instructional  activities 
have  been  specified.  No  change  in  the  current  models  for  matching  media  to 
activities  is  appropriate  until  activities  are  more  widely  specified,  and  these  models 
can  be  tested. 


Grouping  and  Sequencing  of  Instruction 

Findings 

1.  Methodology  -  Universal  principles  for  grouping  and  sequencing  instructional  objec¬ 
tives  do  not  exist,  other  than  that  dependent  objectives  be  placed  later  than  those 
on  which  they  depend.  Alternate  theories  and  conflicting  strategies  abound,  with 
no  knowledge  base  for  resolving  them.  Because  systematically  related  principles 
for  promoting  learning  are  lacking,  sequencing  instruction  must  be  left  to  the 
individual  judgment  of  the  training  developers. 

2.  Application  -  Practices  in  grouping  and  sequencing  instruction  generally  give  no  con¬ 
sideration  to  learning  requirements  other  than  the  order  imposed  by  obvious  depend¬ 
encies.  Attention  is  given  to  constraints  of  non-leaming  factors  such  as  equipment 
availability  and  scheduling. 
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Recommendation 


1.  Organized  and  systematically  related  principles  of  learning  on  which  to  base  grouping 
and  sequencing  decisions  are  not  available.  The  development  of  such  information 
falls  within  the  psychology  of  learning  and  is  beyond  the  scope  of  recommendations 
in  this  report.  In  the  absence  of  such  information,  no  change  in  present  practices 
is  recommended.  jjP 


Development  of  Plan  for 
Authoring  and  Managing  Instruction 


Findings 

1.  Methodology  -  All  models  require  the  preparation  of  a  plan  for  authoring  and 
managing  instruction. 

2.  Application  -  Plans  that  record  course  content  are  often  prepared. 

3.  Application  -  Plans  typically  do  not  specify  instructional  events  and  are  rarely  used 
to  develop  instructional  materials. 

Recommendation 

1.  Explicit  decision  rules  for  selecting  instructional  activities  do  not  presently  exist 
(see  Classification  of  Objectives  and  Selection  of  Instructional  Activities  above). 
Until  such  rules  are  available  and  generally  acknowledged  as  valid,  a  requirement  to 
specify  instructional  activities  in  an  authoring  and  managing  plan  is  likely  to  be 
viewed  as  a  pointless  exercise.  No  change  in  present  practice  is  recommended. 


Review  and  Selection  of  Existing  Materials 


Findings 

1.  Methodology  -  The  ITRO  and  Marine  Corps  models  state  that  decisions  to  use 
existing  materials  (rather  than  author  new  instruction)  are  to  be  based  on  the 
appropriateness  of  these  materials  to  the  previously  specified  characteristics  of 
objectives,  methods,  and  media.  The  AFP  50-58  model  does  not  provide  guidance 
for  reviewing  and  selecting  existing  material. 

2.  Application  -  Characteristics  to  be  identified  in  judging  the  appropriateness  of 
existing  training  materials  are  not  specified. 

3.  Application  -  Review  ■and  use  of  existing  training  materials  is  minimal,  except  for 
those  in  a  course  that  is  being  revised. 

Recommendation 

1.  The  specification  of  necessary  properties  of  materials  for  particular  training  situa¬ 
tions,  and  the  description  and  cataloging  of  existing  materials  to  permit  the  inter¬ 
change  of  matching  components  across  courses,  represent  a  degree  of  perfection 
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that  is  not  presently  attainable.  An  attempt  to  reduce  the  review  and  selection  of 
existing  materials  to  a  systematic  procedure  is  to  act  as  if  the  methods  of  a  well- 
developed  technology  were  available  in  an  area  in  which  judgment  is  in  fact  the 
dominant  factor.  No  change  in  present  practice  is  recommended. 


Authoring  of  Instruction 


Findings 

1.  Methodology  -  All  models  specify  that  instructional  materials  undergo  tryout  during 
the  authoring  process.  The  ITRO  and  Marine  Corps  models  emphasize  that  instruc¬ 
tion  should  be  lean  to  insure  the  economies  of  minimal  instruction. 

2.  Application  -  There  is  little  awareness  of  the  concept  of  lean  instruction,  and 
few  attempts  to  develop  it. 

3.  Application  -  Instruction  is  rarely  given  tryout  and  revision  during  authoring. 

Recommendation 

1.  Training  managers  should  receive  guidance  on  the  purpose  and  importance  of 
developing  lean  instruction.  Guidance  should  indicate  the  role  of  tryout  and 
revision  of  instruction  as  a  necessary  element  of  this  strategy. 


Validation  of  Instruction 


Findings 

1.  Methodology  -  All  models  specify  satisfactory  procedures  for  validating  instruction. 
The  adequacy  of  training  materials  for  attaining  objectives  is  verified  through  the 
administration  of  achievement  tests. 

2.  Application  -  Validation  criteria— that  is,  evidence  that  instruction  is  satisfactory- 
are  rarely  specified. 

3.  Application  -  Instruction  is  rarely  validated  before  it  is  implemented.  When  vali¬ 
dation  does  occur,  it  is  training  materials  (e.g.,  textbooks,  tape/slide  programs) 
that  are  evaluated;  instructor  lessons  and  lesson  plans  are  almost  never  evaluated. 

Recommendation 

1.  Of  the  three  major  types  of  ISD  evaluation  (validation,  internal  evaluation,  external 
evaluation),  validation  has  the  greatest  potential  for  effecting  improvements  in 
instruction.  Once  instructional  materials  have  been  produced  and  instruction  has 
been  implemented,  changes  are  less  likely  to  be  introduced,  and  new  materials  are 
more  difficult  to  generate.  Validation  trials  to  meet  specified  criteria  should  be 
required  before  new  instruction  is  approved. 


101 


Internal  Evaluation 


Findings 

1.  Methodology  -  All  models  specify  adequate  procedures  for  the  internal  evaluation 
of  training.  Quality  control  of  the  training  product  is  to  be  maintained  through 
the  administration  of  objective-referenced  achievement  tests. 

2.  Application  -  Evaluation  and  revision  of  instruction  based  on  needs  revealed  in 
student  performance  (product  evaluation)  are  generally  not  done. 

3.  Application  -  Training  design  decisions  are  rarely  documented  (process  evaluation), 
to  facilitate  redesign  when  instruction  is  found  to  be  inadequate. 

Recommendation 

1.  Trainers  should  be  required  to  determine  and  record  trainee  performance  for  each 
objective.  Although  absolute  standards  to  identify  when  training  revision  is  needed 
are  difficult  to  establish,  the  recording  of  specific  trainee  performance  would  pro¬ 
vide  a  desirable  prerequisite  to  any  revision.  Moreover,  it  would  suggest  relative 
standards  for  the  need  to  revise  training. 


External  Evaluation 


Findings 

1.  Methodology  -  the  ITRO  model  prescribes  the  most  reliable  and  most  costly  way 
to  measure  the  adequacy  of  the  instructional  design  process:  administering  Job 
Performance  Measures  to  graduates  in  the  field.  It  also  provides  the  most  guidance 
for  isolating  causes  of  performance  discrepancies  after  the  external  evaluation  of 
training.  If  summary  evaluations  are  to  be  used,  the  ITRO  and  AFP  50-58  models 
specify  that  information  be  obtained  at  the  task  level  of  specificity,  while  the 
Marine  Corps  model  does  not. 

2.  Methodology  -  None  of  the  models  specify  criteria  that  should  be  used  to  determine 
whether  training  is  to  be  revised  as  a  result  of  external  evaluation,  nor  how  to 
arrive  at  such  criteria.  None  tell  how  good  job  performance  must  be  to  indicate 
that  training  is  acceptable. 

3.  Application  -  The  effectiveness  of  training  is  virtually  never  evaluated  by  the  admin- 
istration  of  Job  Performance  Measures  to  job  incumbents. 

4.  Application  -  Supervisor  summary  evaluations  of  job  incumbent  performance  are 
occasionally  obtained  but  usually  are  not  provided  at  a  task  or  training  objective 
level  of  specificity.  Even  when  performance  and  job  requirements  information  is 
obtained,  it  is  rarely  used  to  redesign  training. 

Recommendations 

1.  The  high  cost  of  administering  performance  tests  to  job  incumbents  and  the  diffi¬ 
culty  of  maintaining  the  necessary  degree  of  objectivity  and  standardization  pre¬ 
clude  their  widespread  use  to  evaluate  training.  No  change  in  present  practice  is 
recommended. 
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2.  Training  can  be  evaluated  by  gathering  information  (rather  than  direct  measurement) 
about  job  performance  and  job  requirements,  at  a  task  and  training  objective  level 
of  specificity,  far  more  thoroughly  than  is  presently  done.  The  failure  to  obtain 
and  use  such  information  is  a  major  shortcoming  in  current  applications  of  ISD. 
While  decision  rules  for  using  such  information  cannot  at  present  be  based  on 
measures  of  total  system  effectiveness,  other  means  for  arriving  at  such  criteria 
are  available.  It  is  recommended  that  operational  commands  define  both  the 
specificity  of  the  task  description  and  the  level  of  performance  they  would  be 
willing  to  use  to  evaluate  the  acceptability  of  job  incumbents.  These  are  the 
criteria  that  supervisors  should  use  to  judge  (rate)  field  performance  and  establish 
the  need  for  training  revision. 


GENERAL  FINDINGS  AND  RECOMMENDATIONS 

Judgments  about  the  adequacy  of  Instructional  System  Development,  its  representa¬ 
tion  in  the  Service  models,  and  the  way  it  is  currently  being  applied  depend  on  one’s 
conception  of  what  ISD  is  and  what  it  is  expected  to  accomplish. 

The  most  general  way  to  define  ISD  is  as  a  means  of  designing  training  to  optimize 
total  system  effectiveness.  Criteria  do  not  exist  for  measuring  total  system  effectiveness; 
ISD  is  not  being  used  to  achieve,  nor  can  it  be  expected  to  result  in,  optimizing  the 
effectiveness  of  the  total  system. 

A  more  circumscribed  view  of  ISD  is  as  a  methodology  for  maximizing  training 
efficiency  within  the  training  subsystem.  However,  information  about  the  effectiveness 
and  costs  of  different  training  strategies  is  far  from  complete,  and  a  trial-and-error 
approach  to  maximizing  efficiency  is  not  practical,  given  the  number  of  possible  combi¬ 
nations  of  methods.  While  ISD  does  provide  a  framework  for  comparing  alternative 
training  strategies,  it  is  not  currently  being  used,  nor  can  it  be  regarded,  as  a  methodology 
for  maximizing  training  efficiency. 

A  more  highly  focused  view  is  that  ISD  is  a  methodology  for  insuring  that  training 
is  relevant  to  the  job.  Its  iterative  and  derivative  character  virtually  assures  that  training 
will  be  relevant  if  available  procedures  are  faithfully  carried  out.  In  practice,  however, 
many  of  its  components  are  omitted,  and  the  close  connection  between  components  that 
is  essential  to  make  the  process  truly  derivative  is  not  maintained.  Most  important,  the 
testing  and  revision  necessary  to  insure  job  relevance  generally  do  not  occur.  The  potential 
of  ISD  to  insure  that  training  meets  job  requirements  is  not  being  realized. 

A  final  conception  of  ISD  is  that  it  is  synonymous  with  the  use  of  modern  training 
techr  logy.  Any  of  the  steps  in  the  training  development  process  are  a  part  of  modem 
training  technology,  and  so  are  any  of  those  particular  training  and  evaluation  methods 
currently  being  emphasized  (e.g.,  self-paced  instruction,  computer-managed  instruction, 
criterion-referenced  testing).  This  definition  of  ISD  is  clearly  the  least  demanding,  since 
in  essence  it  holds  that  undertaking  any  training  development  step  or  using  any  such 
training  or  evaluation  method  constitutes  ISD.  It  is  the  definition  that  is  most  adequately 
represented  in  current  applications  of  ISD. 

In  summary,  then,  two  effects  of  ISD  are  currently  possible:  insuring  that  training 
meets  job  requirements  and  promoting  the  use  of  modem  training  technology.  The  former, 
which  is  clearly  the  more  desirable,  is  not  being  achieved.  The  ISD  modes  does  provide 
the  methodology  for  making  training  relevant,  but  the  mere  existence  of  the  model  does 
not  compel  trainers  to  follow  it.  Trainers  are  relatively  free,  within  fairly  broad  limits. 
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to  determine  the  extent  to  which  they  will  conform  to  the  ISD  process  and  actually  use 
its  products  in  designing  training.  For  example,  front-end  analysis  may  be  undertaken, 
but  its  results  can  be  and  frequently  are  ignored.  Training  objectives  often  are  devel¬ 
oped  with  indifference  to,  or  in  ignorance  of,  actual  task  requirements.  Many  tests  and 
instructional  materials  are  developed  without  regard  to  training  objectives.  Information 
about  the  performance  of  trainees,  once  collected,  is  often  not  used  to  revise  training, 
and  feedback  about  graduates,  if  initiated,  is  often  not  acted  upon. 

Because  ISD  is  a  process,  it  is  difficult  to  observe  directly.  The  occurrence  of  a 
process  is  generally  inferred  from  the  presence  or  absence  of  its  products.  In  the  case 
of  ISD,  however,  the  mere  existence  of  its  products— job  task  lists,  training  objectives, 
achievement  tests,  and  the  like— does  not  by  itself  indicate  that  they  have  been  used  in 
training  development.  Considerable  evidence  that  many  ISD  products  remain  unused 
leads  to  the  conclusion  that,  while  the  generation  of  these  products  can  be  mandated, 
the  ISD  process  and  the  appropriate  use  of  the  products  during  training  development 
cannot. 

Similarly,  a  routine  allocation  of  responsibilities  in  ISD  does  not  necessarily  guar¬ 
antee  that  these  responsibilities  will  be  recognized,  accepted,  and  carried  out.  For  example, 
ISD  methodology  requires  developers  to  specify  the  way  in  which  proficiency  will  be 
developed  or  mediated  for  all  job  tasks— immediately,  through  entry  training;  later, 
through  advanced  training  or  job  experience;  or  through  direct  support  of  job  performance 
by  procedural  aids  or  other  means.  Yet  this  requirement  is  frequently  ignored.  ISD  is 
generally  conducted  to  develop  training  for  only  one  particular  setting,  and  the  manner  in 
which  skill  will  be  acquired  for  tasks  that  are  not  selected  for  training  in  that  setting  is 
usually  not  specified.  Even  where  skills  were  explicitly  identified  for  later  acquisition, 
the  present  study  found  little  evidence  either  that  means  were  developed  for  subsequent 
training  or  that  operational  units  were  informed  of  their  responsibilities  for  insuring  that 
these  skills  be  acquired.  (This  observation  is  based  on  training  development  within  the 
training  subsystem.  The  present  study  did  not  examine  training  developed  or  conducted 
in  operational  units.  Possibly  such  an  examination  would  indicate  that  training  for  job 
skills  in  these  units  is  being  conducted  in  a  more  comprehensive  manner  than  was  sug¬ 
gested  by  the  evidence  in  the  present  study.) 

The  current  failure  of  ISD  applications  to  insure  that  training  meets  job  requirements, 
then,  is  largely  due  not  to  inadequacies  in  the  methodology,  but  to  omissions  and  to 
failure  to  use  its  products  in  a  way  that  makes  the  process  truly  derivative.  An  implica¬ 
tion  of  these  findings  is  that  future  efforts  to  implement  ISD  should  concentrate  on 
finding  ways  to  maintain  the  integrity  of  the  model. 

The  findings  of  the  study  do  not  of  themselves  indicate  how  to  assure  rigorous 
adherence  to  what  is  clearly  a  very  demanding  model,  but  they  do  suggest  that  intro¬ 
ducing  changes  solely  within  the  training  subsystem  is  not  likely  to  have  any  great  effect. 
The  data  strongly  suggest  the  need  for  checks  and  balances  to  guard  against  omissions  in 
the  ISD  process  and  failures  to  use  the  ISD  products.  It  would  appear  logical,  therefore, 
to  provide  for  an  expanded  role  by  operational  commands— the  party  directly  affected 
by  shortcomings  in  training,  and  best  able  to  assess  the  effects  of  training. 

Such  an  expanded  role  for  operational  commands  is,  in  fact,  implied  by  the  model 
itself.  The  derivative  and  iterative  aspects  of  ISD  depend  on  feedback  and  exchange  of 
information  between  trainers  and  users.  It  is  difficult  to  see  how  ISD  could  be  more 
rigorously  applied  unless  such  an  exchange  takes  place.  In  principle,  the  training  sub¬ 
system  seeks  information  about  field  requirements  and  performance  as  the  foundation  on 
which  training  is  constructed.  In  practice,  however,  the  study  found  that  the  training 
subsystem  does  not  have  this  basic  orientation,  often  giving  insufficient  attention  to  the 
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effective  use  of  this  information.  It  is  reasonable  to  suggest  that  a  balanced  relationship- 
one  that  fosters  active  participation  by,  and  communication  between,  operational  and 
training  commands— is  essential  to  the  ISD  process. 

The  following  recommendations  define  the  means  by  which  operational  commands 
can  assume  a  greater  role.  Under  these  recommendations,  operational  commands  would 
participate  to  the  greatest  extent  in  those  parts  of  ISD  where  job  performance  is  repre¬ 
sented  and  where  job  performance  requirements  are  translated  into  training  requirements. 
Those  parts  of  ISD  concerned  with  the  design  of  instructional  strategies  to  meet  train¬ 
ing  and  job  requirements  would  remain  the  province  of  the  training  community. 

It  is  recommended: 

(1)  That  job  requirements  (skills  and  knowledge  required  for  successful  job 
performance)  be  jointly  defined ,  and  agreed  to,  by  training  and  opera¬ 
tional  commands.  This  recommendation  is  a  prerequisite  to  Recommenda¬ 
tion  2,  and  to  all  subsequent  training  development  activities.  If  successful 
dialogue,  negotiation,  and  agreement  are  to  follow,  training  requirements 
must  be  based  on  a  mutually  agreed-upon  definition  of  job  requirements. 

(2)  That  training  requirements  (skills  and  knowledge  to  be  available  at  the 
conclusion  of  training)  be  jointly  defined,  and  agreed  to,  by  training 

and  operational  commands.  Such  an  agreement  should  identify  the  specific 
tasks  and  standards  to  which  proficiency  will  be  developed,  and  should 
delineate  the  respective  responsibilities  of  the  two  parties.  This  includes 
providing  a  means  for  bringing  job  incumbents  to  the  desired  level  of 
proficiency  whenever  agreed-upon  training  requirements  do  not  match  job 
requirements. 

(3)  That  operational  commands  be  required  to  evaluate  the  performance  of 
training  graduates,  and  report  their  findings  to  the  training  commands. 
Unless  operational  commands  evaluate  performance,  feedback  from  users 
to  trainers  will  not  have  a  sound  basis.  Without  reliable  information  about 
the  effects  of  training,  specification  of  training  requirements  will  not 
serve  its  purpose. 

(4)  To  implement  Recommendation  3,  that  task-specific  criteria  for  evaluating 
the  performance  of  graduates,  including  methods  and  standards  to  be 
employed,  be  jointly  defined,  and  agreed  to,  by  training  and  operational 
commands.  Evaluation  criteria  should  be  at  the  task  level  of  specificity 
to  permit  clear  and  useful  diagnosis  of  training.  More  general  evaluations 
are  of  little  use  in  isolating  the  causes  of  inadequate  performance.  Eval¬ 
uation  criteria  must  be  jointly  agreed  to  if  the  results  of  evaluations  are 
to  be  accepted  as  valid,  and  acted  upon. 

Giving  the  operational  commands  a  greater  role  in  both  establishing  training  require¬ 
ments  and  determining  whether  requirements  have  been  met  will  not  of  itself  guarantee 
that  ISD  procedures  will  be  rigorously  applied  to  the  development  of  training.  It  will, 
however,  increase  the  involvement  of  those  who  have  the  most  fundamental  interest  in 
seeing  that  training  has  been  adequately  designed  and  conducted. 

So  long  as  training  development  and  evaluation  are  regarded  as  a  separate  activity 
of  the  training  commands,  there  is  little  reason  to  expect  that  ISD  will  be  applied  any 
more  effectively  than  under  the  present  conditions.  If  training  and  operational  commands 
share  these  activities— each  making  its  specialized  contribution  to  complement  the  work 
of  the  other— the  potential  of  the  ISD  process  for  improving  training  will  be  enhanced. 


Appendix  A 

ISD  ACTIVITIES  QUESTIONNAIRE 
(MAIL  SURVEY  INSTRUMENT) 


ISD  ACTIVITIES  QUESTIONNAIRE 


You  are  being  asked  to  complete  this  questionnaire  to  provide  information  about  courses/ 
training  programs  where  Instructional  System  Development  (ISD)  procedures  have  been 
applied.  This  information  will  be  used  in  the  first  phase  of  a  study  of  ISD  in  the  Armed 
Forces.  In  a  later  phase,  visits  will  be  made  to  selected  installations  to  obtain  more 
detailed  information. 

In  addition  to  identifying  problems  that  have  arisen  in  applying  ISD,  the  study  seeks  to 
identify  those  factors  associated  with  both  successful  and  unsuccessful  applications.  This 
information  will  be  used  to  determine  what  organizational  and  other  support  is  required  for 
ISD  implementation. 

Instructions  for  Completing  the  Questionnaire 

1.  Enter  identification  information  below. 

Organization: _ _ _ _ 

(Command  or  School  or  Unit) 

Address:  _ _ _ _ 


Person  to  be  contacted  for  additional  information: _ 

Commercial  phone:  _ _ 

2.  Enter  below  the  total  number  of  courses/training  programs  administered  by  the  above 
organization.  For  the  purpose  of  this  questionnaire,  “courses/training  programs”  include 
any  course  or  program  of  instruction,  training,  or  education. 

Total  number  conducted  at  the  school  or  unit:  _ 

Total  number  conducted  by  correspondence  or  extension:  _ 

3.  On  the  following  page,  list  the  titles  of  all  courses/training  programs  for  which  job  analysis 
data  (task  lists)  have  been  compiled  and  are  currently  available.  Although  this  questionnaire 
does  not  require  ISD  to  be  defined  in  terms  of  any  particular  set  or  sequence  of  actions, 

it  does  require  that  a  list  of  job  tasks  be  available.  Therefore,  list  only  courses/training 
programs  for  which  task  lists  are  available. 

4.  For  each  course /training  program  that  has  been  listed,  provide  information  about  ISD 
activities  by  completing  columns  2-22  on  the  following  pages. 

Human  Resources  Research  Organization 
(Contract  903-77-C-033) 

Office  of  Assistant  Secretary  of  Defense 
(Manpower,  Reserve  Affairs,  &  Logistics) 
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Instructions  for  Completing  Columns  1  -  23 


Column  1 :  Enter  Courses/T raining  Programs  for  which  job  analysis  data  (task  lists)  are  available. 


Column  2: 


Column  3: 

Columns  4  - 
22: 


Enter  N  if  the  job  analysis  data  were/are  being  used  to  develop  a  new  course. 

Enter  E  if  the  job  analysis  data  were/are  being  used  to  evaluate/revise  an  existing  course. 

Enter  0  if  the  job  analysis  data  were/a  re  being  used  for  other  purposes  than  course  development 
or  revision. 

If  a  course  was/is  being  developed  or  revised,  enter  the  approximate  date  when  the  first  graduates 
reached/will  reach  the  job. 

Provide  information  about  ISD  activities 

Enter  A  if  . . .  the  activity  was  undertaken,  and  documentation  or  products  are  available. 

Enter  P  if  . . .  the  activity  was  attempted,  but  products  are  not  available  due  to  methodology 
or  other  problems. 

Enter  N/C  if . .  .  the  activity  is  being  undertaken,  but  is  not  completed. 

Enter  N/A  if  . . .  the  activity  is  not  applicable  or  was  not  undertaken. 


Approximate  Date  when  first  graduates 
reached  (or  will  reach)  the  job. 

N  =  New  Course 
E  =  Existing  Course 
0  *  Other  Purpose(s) 


Analyzed  problem/systtm/existing  course  to  determine 
need  for  training  development. 

Obtained  priority  information  about  tasks  (frequency, 
delay  tolerance,  criticality,  etc.) 

Analyzed  tasks  into  elements,  conditions,  and 
standards. 


COURSES/ 

TRAINING  PROGRAMS 
FOR  WHICH 
JOB  ANALYSIS  DATA 
ARE  AVAILABLE 


Selected  tasks  for  training  on  the  basis 
of  task  priority  information. 

Clustered  tasks  and  selected  instruc¬ 
tional  setting  on  the  basis  of  priority 
information,  resources,  and  output 
requirements. 

Developed  terminal  and  inter¬ 
mediate  learning  objectives. 

Sequenced  and  clustered 
objectives. 


2  3 


8  9 
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Appendix  B 

INCIDENCE  BY  SERVICE  OF  PREREQUISITES  AND 
PROCEDURES  IN  COURSES  EXAMINED 


i 

! 


( 

i 


i 


Appendix  B 


Incidence  by  Service  of  Prerequisites  and 
Procedures  in  Courses  Examined1 


present/  Absent/  Not  Applicable/ 

Accomplished  Not  Accomplished  Not  Determined 

Need/Discrepancy  Analysis 


Prerequisites 


1 .  Information  is  available  about 

Army 

2 

0 

5 

possible  discrepancies  between 

Navy 

11 

1 

11 

training  and  field  requirements. 

Marine  Corps 

2 

0 

6 

Air  Force 

16 

1 

3 

Procedures 

1.  Analyze  initial  and  supple¬ 

Army 

2 

5 

0 

mentary  information. 

Navy 

10 

10 

3 

Marine  Corps 

2 

2 

3 

Air  Force 

10 

5 

5 

2.  Identify  and  specify 

Army 

2 

5 

0 

discrepancy. 

Navy 

9 

11 

3 

Marine  Corps 

1 

2 

4 

Air  Force 

11 

5 

4 

3.  Specify  ISO  entry  point 

Army 

1 

6 

0 

and  boundaries  of  redesign 

Navy 

8 

12 

3 

process. 

Marine  Corps 

1 

3 

3 

Air  Force 

9 

6 

5 

Identification  of  Job  Requirements 

Prerequisites 

None 

Procedures 

1.  Construct  provisional  task  list.  Army  7  0  0 

Navy  16  4  3 

Marine  Corps  6  0  1 

Air  Force  9  9  2 

( Continued) 

'Total  number  of  courses  examined  in  each  Service:  Army-7:  Navy-23;  Marine  Corps-7;  Air  Force— 20. 
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1 


Identif  ication  of  Job  Requirements 

(Continued) 


Present/ 

Accomplithed 


Absent/ 

Not Accomplithed 


Verify  and  revise  on  basis  of 

Army 

6 

1 

review/survey  of  job 

Navy 

11 

8 

incumbents. 

Marine  Corps 

4 

1 

Air  Force 

6 

10 

Collect  task  priority 

Army 

5 

1 

information. 

Navy 

7 

12 

Marine  Corps 

4 

0 

Air  Force 

8 

10 

Selection  of  Tasks  for  Training 


Prerequisites 


1.  Tasks  performed  in  the 

Army 

7 

0 

job  are  listed. 

Navy 

15 

4 

Marine  Corps 

7 

0 

Air  Force 

9 

9 

2.  information  is  available 

Army 

5 

2 

for  establishing  importance 

Navy 

7 

11 

of  tasks  and  need  for 

Marine  Corps 

2 

1 

training. 

Air  Force 

9 

9 

3.  Decision  rules  to  be  applied 

Army 

3 

4 

to  task  information  are 

Navy 

2 

17 

available. 

Marine  Corps 

1 

2 

Air  Force 

1 

17 

Procedures 

1.  Apply  decision  rules  to 

Army 

2 

5 

information  for  each  task 

Navy 

1 

18 

to  determine  training 

Marine  Corps 

1 

1 

priorities. 

Air  Force 

1 

17 

2.  Select  tasks  for  training 

Army 

2 

5 

on  basis  of  training 

Navy 

2 

17 

priorities  and  resource 

Marine  Corps 

1 

1 

availability. 

Air  Force 

1 

17 

(Continued) 


Not  Applicable/ 
Not  Determined 


0 

4 

2 

4 

1 

4 

3 

2 


0 

4 
0 
2 

0 

5 
4 
2 

0 

4 

4 

2 


0 

4 

5 
2 

0 

4 

5 
2 
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Present  /  Absent/ 

Accomplished  Not  Accomplished 

Analysis  of  Tasks 

Prerequisites 


1 .  Tasks  selected  for  training 

Army 

7 

0 

are  listed. 

Navy 

16 

15 

Marine  Corps 

3 

2 

Air  Force 

15 

3 

Procedures 


1 .  For  each  task  specify  for 
the  job  environment: 


—  Conditions  of  performance. 

Army 

7 

0 

Navy 

3 

15 

Marine  Corps 

4 

0 

Air  Force 

2 

14 

—  Behavioral  elements. 

Army 

6 

0 

Navy 

9 

9 

Marine  Corps 

4 

0 

Air  Force 

4'  12 

4 

—  Standards  of  performance. 

Army 

7 

0 

Navy 

5 

14 

Marine  Corps 

4 

0 

Air  Force 

3 

14 

Construction  of  Job  Performance 
Measures 


Prerequisites 


1 .  Tasks  selected  for  training 

Army 

4 

0 

are  listed. 

Navy 

3 

1 

Marine  Corps 

1 

0 

Air  Force 

4 

1 

2.  Training  requirements  for 

Army 

3 

1 

these  tasks  have  not  been 

Navy 

2 

2 

identified. 

Marine  Corps 

1 

0 

Air  Force 

2 

3 

Procedures 

1.  Construct  a  test  for  measuring 

Army 

4 

3 

the  performance  of  each  task 

Navy 

2 

17 

selected  for  training. 

Marine  Corps 

0 

4 

Air  Force 

5 

12 

( Continued ) 


Not  Applicable/ 
Not  Determined 


0 

2 

2 

2 


0 

5 

3 

4 

1 

5 

3 

4 

0 

4 

3 

3 


3 

19 

6 

15 

3 

19 

6 

15 


0 

4 

3 

3 
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I 


4 
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Construction  of  Job  Performance 

Measures  (Continued) 

2.  Validate  each  Job  Performance  Army 

Measure  to  insure  that  it  p re-  Navy 

diets  task  performance.  Marine  Corps 

Air  Force 


Selection  of  Setting 


Prerequisites 


1 .  Tasks  can  be  assigned  to 
any  of  several  settings. 

Army 

Navy 

Marine  Corps 
Air  Force 

2.  Information  is  available 
on  costs  of  training  in 
different  settings. 

Army 

Navy 

Marine  Corps 
Air  Force 

3.  Information  is  available 
on  effects  of  training  in 
different  settings  on  total 
system  effectiveness. 

Army 

Navy 

Marine  Corps 
Air  Force 

Procedures 

1 .  Assign  each  task  or 
group  of  tasks  to  its 
appropriate  setting. 

Army 

Navy 

Marine  Corps 
Air  Force 

Development  of  Training  Objectives 
and  Objectives  Hierarchies 

Prerequisites 

1 .  Tasks  selected  for  training 
are  listed. 

Army 

Navy 

Marine  Corps 
Air  Force 

2.  Information  is  available 
about  training  constraints 
that  make  it  necessary  to 
modify  task  requirements. 

Army 

Navy 

Marine  Corps 
Air  Force 

Present/ 

Absent/ 

Not  Applicable/ 

Accomplished 

Not  Accomplished 

Not  Determined 

0 

4 

3 

2 

17 

4 

0 

5 

2 

1 

12 

7 

4 

0 

3 

8 

10 

5 

0 

2 

5 

6 

11 

3 

0 

0 

7 

0 

1 

22 

0 

0 

7 

0 

2 

18 

0 

0 

7 

0 

1 

22 

0 

0 

7 

0 

2 

18 

3 

4 

0 

7 

15 

1 

0 

6 

1 

6 

12 

2 

7 

0 

0 

17 

4 

2 

5 

0 

2 

15 

3 

2 

0 

0 

7 

0 

0 

23 

0 

0 

* 

7 

0 

0 

20 

(Continued) 
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m*jk)Lx 


Present/ 

Accomplished 


Absent/ 

Not  Accomplished 


Not  Applicable/ 
Not  Determined 


Development  of  Training  Objectives  and 
Objectives  Hierarchies  (Continued) 


3.  Information  is  available  about 

Army 

how  modification  of  task 

Navy 

requirements  will  affect 

Marine  Corps 

training  efficiency. 

Air  Force 

4.  Estimates  of  capabilities 

Army 

of  entering  trainees  for 

Navy 

learning  and  performing 

Marine  Corps 

each  task  are  available. 

Air  Force 

Procedures 

1 .  Specify  task  requirements 

Army 

(behaviors,  conditions, 

Navy 

standards)  for  training. 

Marine  Corps 

Air  Force 

2.  Perform  hierarchical  anal¬ 

Army 

ysis  of  tasks  to  identify 

Navy 

intermediate  training 

Marine  Corps 

objectives. 

Air  Force 

Development  of  Achievement  Tests 


Prerequisites 


1.  Training  objectives  have 

Army 

6 

0 

been  specified. 

Navy 

21 

0 

Marine  Corps 

O 

0 

Air  Force 

18 

0 

2.  Instructional  materials  have 

Army 

3 

3 

not  been  developed. 

Navy 

8 

12 

Marine  Corps 

2 

1 

*\ir  Force 

3 

v  1S 

Procedures 

1.  Determine  appropriate 

Army 

0 

3 

types  of  tests  based  on 

Navy 

0 

1 

characteristics  of  objectives. 

Marine  Corps 

0 

2 

Air  Force 

0 

0 

2.  Construct  tests  to  assess 

Army 

6 

0 

attainment  of  objectives. 

Navy 

0 

18 

Marine  Corps 

4 

0 

Air  Force 

17 

0 

(Continued) 
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4 
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Identification  of  Entry  Behavior 


Prerequisites 


1. 

Objectives  have  been  derived 

Army 

through  hierarchical  anal¬ 

Navy 

ysis  of  tasks. 

Marine  Corps 
Air  Force 

2. 

Tests  are  available  to 

Army 

measure  objectives. 

Navy 

Marine  Corps 
Air  Force 

Procedures 

1. 

Identify  sample  that  is 

Army 

representative  of  trainees. 

Navy 

Marine  Corps 
Air  Force 

2. 

Administer  tests  to  sam¬ 

Army 

ple,  and  determine  accu¬ 

Navy 

racy  of  earlier  estimate  of 

Marine  Corps 

entry  behavior. 

Air  Force 

3. 

Add  or  delete  objectives 

Army 

as  indicated  by  test 

Navy 

results,  and  repeat  cycle. 

Marine  Corps 
Air  Force 

Present/  Absent/ 

Accomplished  Not  Accomplished 


3 

5 

2 

1 


3 

15 

4 

16 


6 

19 

3 

17 


0 

0 

0 

0 


0 

0 

0 

0 


6 

20 

6 

17 


0 

D 

0 

0 


6 

20 

6 

17 


0 

0 

0 

0 


6 

2a 

6 

17 


Classification  of  Objectives  and 

Selection  of  Instructional  Activities 


Prerequisites 


1 .  Information  is  available  about 

Army 

0 

0 

types  of  instructional  activ¬ 

Navy 

0 

0 

ities  appropriate  to  acquiring 

Marine  Corps 

0 

0 

different  types  of  capabilities. 

Air  Force 

0 

0 

Procedures 

1 .  Classify  each  objective  or 

Army 

2 

3 

group  of  objectives  accord¬ 

Navy 

4 

16 

ing  to  type  of  capability. 

Marine  Corps 

0 

5 

Air  Force 

2 

16 

2.  Specify  instructional  activ¬ 

Army 

0 

6 

ities  for  each  objective  accord¬ 

Navy 

1 

19 

ing  to  type  of  capability. 

Marine  Corps 

0 

5 

Air  Force 

0 

18 

Not  Applicable/ 
Not  Determined 


1 

3 

1 

3 

1 

4 
4 

3 


1 

3 

1 

3 

1 

3 

1 

3 

1 

3 

1 

3 


7 

23 

7 

20 


2 

3 

2 

2 

1 

3 

2 

2 


(Continued) 
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Present/ 

Accomplished 

Absent/ 

Not  Accomplished 

Not  Applicable/ 
Not  Determined 

Selection  of  Instructional  Methods 

Prerequisites 

1.  Setting  has  been  specified. 

Army 

6 

0 

1 

Navy 

22 

0 

1 

Marine  Corps 

6 

0 

1 

Air  Force 

18 

0 

2 

2.  Trainee  characteristics 

Army 

0 

0 

7 

have  been  identified. 

Navy 

0 

1 

22 

Marine  Corps 

0 

0 

7 

Air  Force 

0 

0 

20 

3.  Instructional  activities 

Army 

0 

6 

1 

have  been  specified. 

Navy 

1 

20 

2 

Marine  Corps 

0 

5 

2 

Air  Force 

0 

18 

2 

4.  Information  is  available 

Army 

0 

0 

7 

on  how  the  costs  and  effec¬ 

Navy 

0 

1 

22 

tiveness  of  alternate  methods 

Marine  Corps 

0 

0 

7 

vary  for  specified  settings. 

Air  Force 

0 

2 

18 

trainee  characteristics,  and 
instructional  activities. 

Procedures 

1 .  Specify  the  methods  of 

Army 

6 

0 

1 

instruction  to  be  employed 

Navy 

22 

0 

1 

for  each  objective  or  group 

Marine  Corps 

5 

0 

2 

of  objectives. 

Air  Force 

18 

0 

2 

Selection  of  Media 

Prerequisites 

1.  Training  designers  are 

Army 

0 

1 

6 

tree  to  select  from  a 

Navy 

3 

5 

15 

range  of  media. 

Marine  Corps 

0 

0 

7 

Air  Force 

3 

3 

14 

2.  Instructional  activities 

Army 

0 

6 

1 

have  been  specified. 

Navy 

4 

16 

3 

Marine  Corps 

0 

5 

2 

Air  Force 

0 

18 

2 

3.  Methods  of  training  have 

Army 

3 

0 

4 

been  specified. 

Navy 

18 

1 

4 

Marine  Corps 

0 

0 

7 

Air  eorce 

18 

0 

2 

( Continued ) 
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Selection  of  Media  (Continued) 

4.  Information  is  available  con¬ 
cerning  the  appropriateness 
of  different  media  to  imple¬ 
ment  different  activities  and 
to  be  used  in  conjunction 
with  different  methods. 

Army 

Navy 

Marine  Corps 
Air  Force 

5.  Information  is  available  con¬ 
cerning  the  costs  of  dif¬ 
ferent  media. 

Army 

Navy 

Marine  Corps 
Air  Force 

Procedures 

1 .  Determine  which  media 
will  be  suitable  to  imple 
ment  the  instructional 
activities. 

Army 

Navy 

Marine  Corps 
Air  Force 

2.  Consider  relative  costs  of 
media  determined  above  and 
select  most  economical  set 
of  media. 

Army 

Navy 

Marine  Corps 
Air  Force 

Grouping  and  Sequencing  of 
Instruction 

Prerequisites 

1 .  Knowledge  of  the  effects  on 
learning  of  different  sequenc¬ 
ing  plans  is  available. 

Army 

Navy 

Marine  Corps 
Air  Force 

Procedures 

1 .  Identify  commonality  of 
subject  matter  and  antici¬ 
pated  transfer  of  learning 
between  objectives. 

Army 

Navy 

Marine  Corps 
Air  Force 

2.  Identify  degree  of  depend¬ 
ency  between  objectives. 

Army 

Navy 

Marine  Corps 
Air  Force 

3.  Select  overall  sequencing 
principle(s). 

Army 

Navy 

Marine  Corps 
Air  Force 

Present/ 

Accomplished 


Absent/ 

Not  Accomplished 


Not  Applicable/ 
Not  Determined 


(Continued) 
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Grouping  and  Sequencing  of 
Instruction  (Continued) 


Present / 
Accomplished 


Absent/  Not  Applicable/ 

Not  Accomplished  Not  Determined 


4.  Group  and  sequence 

Army 

6 

0 

1 

objectives. 

Navy 

21 

0 

2 

Marine  Crops 

6 

0 

1 

Air  Force 

18 

0 

2 

Development  of  Plan  for  Authoring 
and  Managing  Information 

Prerequisites 

1.  Objectives,  instructional 

Army 

0 

6 

,  1 

activities,  methods,  and 

Navy 

5 

15 

3 

media  have  all  been  specified. 

Marine  Corps 

0 

5 

2 

Air  Force 

0 

18 

2 

Procedures 

1 .  Specify  the  content  and 

Army 

3 

2 

2 

design  of  each  lesson. 

Navy 

13 

6 

4 

Marine  Corps 

1 

3 

3 

Air  Force 

14 

4 

2 

2.  Specify  how  the  instruction 

Army 

6 

0 

1 

will  be  conducted  and 

Navy 

21 

0 

2 

managed. 

Marine  Coips 

6 

0 

1 

Air  Force 

17 

1 

2 

Review  and  Selection  of 

Existing  Materials 

Prerequisites 

1 .  Instructional  activities  have 

Army 

0 

5 

2 

been  specified. 

Navy 

5 

15 

3 

Marine  Corps 

0 

5 

2 

Air  Force 

0 

18 

2 

2.  Methods  of  training  have 

Army 

5 

0 

2 

been  specified. 

Navy 

20 

0 

3 

Marine  Corps 

6 

0 

1 

Air  Force 

18 

0 

2 

3.  Media  have  been  specified. 

Army 

3 

2 

2 

Navy 

16 

3 

4 

Marine  Corps 

2 

0 

5 

Air  Force 

16 

2 

2 

(Continued) 


Review  end  Selection  of 

Existing  Materials  f Continued ) 


Present  /  Absent/ 

Accomplished  Not  Accomplished 


Procedures 


1 .  Examine  existing  instructional 

Army 

1 

4 

materials  to  determine  whether 

Navy 

5 

14 

any  meet  the  specifications 

Marine  Corps 

0 

5 

for  instructional  activities, 
methods,  and  media. 

Air  Force 

4 

13 

2.  Select  materials  or  parts 

Army 

1 

4 

of  materials  that  meet  the 

Navy 

6 

13 

specifications,  or  which  could 

Marine  Corps 

0 

5 

be  efficiently  revised. 

Air  Force 

4 

13 

Authoring  of  Instruction 

Prerequisites 

1 .  Objectives  have  been 

Army 

5 

0 

grouped  and  sequenced. 

Navy 

21 

0 

Marine  Crops 

6 

0 

Air  Force 

17 

0 

2.  Instructional  activities 

Army 

0 

5 

have  been  specified. 

Navy 

5 

15 

Marine  Corps 

0 

5 

Air  Force 

0 

17 

3.  Methods  of  training  have 

Army 

5 

0 

been  specified. 

Navy 

20 

0 

Marine  Corps 

6 

0 

Air  Force 

17 

0 

4.  Media  have  been  specified. 

Army 

3 

2 

Navy 

16 

2 

Marine  Corps 

2 

3 

Air  Force 

16 

1 

5.  Lesson  structure  and 

Army 

2 

2 

content  have  been  planned. 

Navy 

14 

4 

Marine  Corps 

1 

4 

Air  Force 

16 

1 

Procedures 

1.  Develop  lean  instruction. 

Army 

1 

3 

Navy 

9 

10 

Marine  Corps 

0 

1 

Air  Force 

1 

14 

(Continued) 


Authoring  of  Instruction  ( Continued ) 


2.  Try  out  instruction  on  small  Army 

number  of  persons  representa-  Navy 

tive  of  students.  Marine  Corps 

Air  Force 

3.  Revise  and  augment  instruc-  Army 

tion  as  necessary.  Navy 

Marine  Corps 


Air  Force 


Present/ 

Accomplished 

Absent/ 

Not  Accomplished 

Not  Applicable/ 
Not  Determined 

2 

3 

2 

7 

12 

4 

0 

2 

5 

1 

14 

5 

1 

3 

3 

3 

11 

9 

0 

2 

5 

1 

14 

5 

Validation  of  Instruction 


Prerequisites 


1.  Objective-referenced 

Army 

5 

0 

2 

achievement  tests  are 

Navy 

20 

0 

3 

available. 

Marine  Corps 

6 

0 

1 

Air  Force 

16 

0 

4 

Procedures 

1.  Specify  achievement  test 

Army 

0 

3 

4 

validation  criteria  Inumber 

Navy 

7 

9 

7 

and  percent  of  persons  in 

Marine  Corps 

0 

3 

4 

validation  sample  required 
to  pass  tests). 

Air  Force 

0 

15 

5 

2.  Specify  additional  valida¬ 

Army 

0 

3 

4 

tion  criteria. 

Navy 

4 

11 

8 

Marine  Corps 

0 

3 

4 

Air  Force 

0 

15 

5 

3.  Present  instruction,  adminis¬ 

Army 

2 

2 

3 

ter  achievement  tests,  analyze 

Navy 

7 

11 

5 

results,  revise  instruction,  and 

Marine  Corps 

0 

3 

4 

repeat  cycle  until  validation 

Air  Force 

0 

15 

5 

criteria  are  met. 


Internal  Evaluation 
Prerequisites 

1.  Records  of  students' perform-  Army  3  0  4 

ance  on  achievement  tests  are  Navy  9  0  14 

available.  Marine  Corps  3  0  4 

Air  Force  13  1  6 

(Continued) 
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Internal  Evaluation  (Continued) 

Prerent/ 

Accomplished 

Absent/ 

Not  Accomplished 

Not  Applicable/ 
Not  Determined 

2.  Documentation  is  available  of 

Army 

0 

3 

4 

what  occurred  during  the  ISD 

Navy 

5 

6 

12 

process,  including  rationales 

Marine  Corps 

0 

3 

4 

for  decisions,  departures  from 

Air  Force 

8 

7 

5 

standard  procedures,  etc. 

Procedures 

1.  Specify  evaluation  criteria 

Army 

0 

3 

4 
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